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1.  Introduction 


The  battlefield  of  the  future  will  use  robotic/unmanned  vehicles  in  increasing  numbers.  Human 
operators  may  find  themselves  overwhelmed  in  the  attempt  to  manage  multiple  robots  while 
simultaneously  performing  tasks,  such  as  local  security  maintenance.  Prior  research  has  shown 
that  operators  who  were  assigned  to  manage  multiple  robots  suffered  performance  degradation 
on  concurrent  tasks  (Chen  et  al.,  2008).  Specifically,  situation  awareness  (SA)  reduces  as  the  size 
of  the  robot  team  increases,  whereas  cognitive  resources  become  depleted  (Chen  and  Bames, 
2012a;  Wang  et  al.,  2008;  Wang  et  al.,  2009),  and  response  times  increase.  The  increase  in 
response  times  is  particularly  evident  when  switching  between  tasks — such  as  switching  from  a 
target  detection  task  to  managing  convoy  vehicle  spacing  (Squire  and  Parasuraman,  2010).  Further, 
when  a  single  operator  manages  multiple  robots  by  interacting  with  them  individually,  workload 
increases  as  the  number  of  robots  increase,  the  number  of  tasks  that  can  be  successfully  completed 
within  a  designated  time  interval  decreases,  and  the  number  of  system  failures  and  accidents 
increase  (Adams,  2009).  Clearly,  it  is  impractical  and  inefficient  for  the  operator  to  interact  with 
each  robotic  team  member  individually  when  the  operator  is  coordinating  the  actions  of  multiple 
robots. 

RoboLeader  is  an  intelligent  agent  developed  under  the  U.S.  Army  Research  Laboratory’s 
(ARL’s)  Director’s  Research  Initiative  and  was  designed  to  assist  a  human  operator  with 
managing  a  team  of  robots  (Chen  and  Barnes,  2012a,  2012b;  Snyder  et  al.,  2010).  Instead  of  the 
human  operator  interacting  with  each  robot  individually,  the  operator  only  interacts  with 
RoboLeader,  thus  freeing  cognitive  resources  for  other  tasks.  The  effectiveness  of  RoboLeader 
was  previously  investigated  in  three  human-in-the-loop  simulation  experiments  and  was  found  to 
improve  operator’s  mission  completion  times,  as  well  as  reduce  their  perceived  workload  (Chen 
and  Bames,  2012a,  2012b). 

1.1  Level  of  Autonomy 

This  study  investigates  the  appropriate  level  of  autonomy  (LOA)  to  assign  to  RoboLeader  in  a 
human-agent  teaming  task.  The  LOA  of  a  system  changes  the  dynamics  of  the  human-agent 
relationship  (Parasuraman  and  Riley,  1997;  Parasuraman  et  al.,  2000),  and  as  discussed  in  Chen 
et  al  (2011),  it  is  important  for  system  designers  to  select  a  LOA  that  helps  the  operator  maintain 
SA  while  keeping  the  operator  engaged  in  the  task.  A  LOA  that  is  too  low  may  increase 
workload,  resulting  in  operator  fatigue  and  decreased  perfonnance.  In  comparison,  a  too  high 
LOA  may  disengage  the  operator,  resulting  in  automation-induced  complacency,  decreased  SA, 
and  vigilance  decrements  (Parasuraman  and  Riley,  1997;  Endsley,  1999;  Parasuraman  et  al., 
2000).  Often  a  medium  LOA  is  recommended  to  reduce  human  out-of-the-loop  problems  and  to 
help  maintain  manual  skills. 
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However,  no  LOA  seems  to  be  effective  at  preventing  errors  of  commission,  as  these  errors  are 
also  caused  by  factors  other  than  complacency,  and  as  such,  tend  to  occur  independent  of  LOA 
(Manzey  et  ah,  2008;  Reichenbach  et  ah,  2010). 

1.2  Multitasking 

It  is  becoming  commonplace  to  expect  an  operator  to  concurrently  conduct  several  tasks.  When 
switching  between  tasks,  the  primary  task  perfonnance  suffers  when  it  is  interrupted  by  a 
secondary  task  (Cummings,  2004;  Monsell,  2003).  Advanced  preparation  for  a  task  switch  may 
reduce  the  impact  associated  with  switching,  although  it  cannot  completely  negate  the  effect 
(Altmann  and  Trafton,  2004).  Although  performance  may  recover  somewhat  after  switching 
tasks,  there  is  a  long-term  negative  effect  on  response  times,  as  well  as  increased  error  rates 
(Monsell,  2003).  Performance,  particularly  in  the  secondary  tasks,  can  be  improved  by  increasing 
LOA  assistance  (Manzey  et  ah,  2008). 

When  switching  between  different  types  of  tasks,  the  associated  costs  of  task  switching  also 
increase  (Squire  et  ah,  2006).  Task  switching  costs  may  be  increased  with  higher  LOAs,  as  these 
can  lead  to  a  less  active  information  processing  mode  by  the  operator  (Chen  and  Bames,  2012a; 
Wickens  et  al.,  2003),  which  forces  the  operator  to  regain  knowledge  of  the  state  of  the  system 
before  conducting  the  new  task  (Squire  and  Parasuraman,  2010).  Interface  design  affects 
switching  costs  as  well,  particularly  when  managing  robot  teams  of  increasing  size  (Squire  et  al., 
2006). 

1.3  Individual  Differences 

The  effects  of  individual  differences  on  operator  performance  were  also  evaluated  in  the  current 
study.  Specifically,  the  effects  of  individual  differences  in  perceived  attentional  control  (PAC), 
spatial  ability  (SpA),  and  gaming  experience  on  the  operators’  robotics  control,  as  well  as 
multitasking  perfonnance,  were  investigated. 

1.3.1  Attentional  Control 

The  relationship  between  perceived  attentional  control  and  multitasking  perfonnance  was 
examined.  Attentional  control  refers  to  an  individual’s  ability  to  choose  what  they  will  attend  to 
and  what  they  will  ignore  (Astle  and  Scerif,  2009).  Although  there  are  individual  differences  in 
multitasking  perfonnance,  some  people  are  less  likely  to  suffer  performance  degradation  while 
multitasking  (Rubinstein  et  al.,  2001).  There  is  evidence  that  individuals  with  higher  PAC  can 
allocate  their  attention  more  flexibly  and  effectively  (Derrybeny  and  Reed,  2002;  Chen  and 
Joyner,  2009).  In  Chen  and  Bames  (2012a),  participants  with  higher  PAC  reported  lower 
workload  than  did  those  with  lower  PAC. 

1.3.2  Spatial  Ability 

Previous  research  has  found  performance  differences  on  certain  tasks  between  individuals  with 
high  and  low  SpA.  Lathan  and  Tracey  (2002)  demonstrated  that  people  with 
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higher  SpA  finished  their  tasks  faster  and  had  fewer  errors  during  a  teleoperation  task  through  a 
maze.  Lathan  and  Tracey  suggested  that  military  missions  could  benefit  from  selecting  personnel 
with  higher  SpA  to  operate  robotic  devices.  Spatial  ability  was  also  found  to  be  a  good  predictor 
of  the  operator’s  robotics  perfonnance  (Chen  et  ah,  2008;  Chen  and  Barnes,  2012a).  In  Chen  and 
Barnes  (2012a),  participants  with  higher  SpA  scanned  the  simulated  robot’s  video  views 
significantly  faster  than  those  with  lower  SpA,  resulting  in  more  targets  detected  and  higher  SA. 

1.3.3  Gaming  Experience 

There  is  evidence  that,  when  compared  to  infrequent  gamers,  frequent  video  game  players 
collaborate  more  with  automated  systems  (Cummings  et  ah,  2010),  show  more  flexibility  and 
efficiency  in  visual  attention  (Green  and  Bavelier,  2003),  and  have  better  visual  memory  (Green 
and  Bavelier,  2006).  Video  gaming  experience  has  also  been  shown  to  correspond  with  faster 
reaction  times,  improved  tracking  of  moving  targets  (McKinley  et  ah,  2011),  better  perfonnance 
in  target  detection  tasks,  and  improved  SA  (Chen  and  Barnes,  2012a,  2012b). 

1.4  Current  Study 

In  the  current  experiment,  we  simulated  a  multitasking  environment  where  the  operator  (i.e., 
vehicle  commander)  had  to  supervise  the  routes  of  three  vehicles  (i.e.,  their  own  manned  ground 
vehicle  [MGV],  an  unmanned  aerial  system  [UAS],  and  an  unmanned  ground  vehicle  [UGV]), 
and  the  distance  separations  among  them  while  maintaining  proper  360°  local  security  around 
their  MGV.  The  U.S.  Army  is  currently  developing  360°  indirect-vision  display  capabilities  to 
enable  vehicle  commanders  to  see  their  immediate  environment  via  streaming  video  sent  from 
cameras  mounted  outside  the  MGV  (Elliott  et  ah,  in  press).  The  three  simulated  vehicles  traveled 
in  an  urban  environment  as  a  convoy,  and  the  participant  had  to  decide  whether  and  how  the 
routes  for  the  convoy  needed  to  change  based  on  intelligence  reports  and/or  environmental 
events  (e.g.,  threats  present,  environmental  hazards/obstacles),  as  well  as  pause/start  vehicle 
movement  in  order  to  maintain  assigned  vehicle  separation.  The  paradigm  followed  that  which 
was  previously  studied  by  Chen  and  Barnes  (2012a)  and  the  participants  performed  the  convoy 
management  duties  either  manually  or  with  assistance  from  RoboLeader  (for  the  vehicle 
separation  task  only  or  for  both  vehicle  separation  and  convoy  route  planning).  The  participants 
also  concurrently  monitored  an  indirect-vision  display  of  the  environment  surrounding  the  MGV 
and  reported  any  threats  present  in  their  immediate  environment.  Threats  were  identified  as 
civilian  persons  or  vehicles  that  were  armed.  Visual  density,  type,  and  number  of  threats  were 
held  constant  across  missions  and  automation  levels. 

1.4.1  Stated  Hypothesis/Objective 

This  experiment  manipulated  the  LOA  of  RoboLeader  to  assist  the  operator  with  his/her  convoy 
management  tasks.  There  were  three  levels  of  automation  assistance:  Manual  (no  RoboLeader 
assistance)  and  two  RoboLeader  levels:  Semi-Autonomous  (maintaining  vehicle  distance 
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/separation  only)  and  Fully  Autonomous  (vehicle  separation  +  route  planning).  The  participants 
also  maintained  360°  local  security  by  monitoring  the  two  180°  indirect-vision  display  camera 
feeds  (one  forward,  one  rearward)  and  detecting  targets  when  they  appeared.  The  effects  of 
individual  differences  in  spatial  ability,  attentional  control,  and  prior  gaming  experience  on  the 
operators’  task  performance  were  also  investigated.  Therefore,  the  principal  objectives  were  (1) 
to  determine  whether  increasing  levels  of  assistance  from  RoboLeader  improved  a  commander’s 
ability  to  maintain  360°  security  and  SA,  (2)  whether  individual  differences  had  differential 
effects  on  the  commander’s  ability  to  maintain  security,  and  (3)  whether  increasing  levels  of 
assistance  from  RoboLeader  resulted  in  reduced  workload  levels  and  improved  scan  efficiency. 
In  addition  to  the  stated  hypotheses,  eye  movement  data  was  collected  to  evaluate  how  well 
physiological  measures  of  workload,  interest,  and  scan  efficiency  correlate  with  performance, 
SA,  and  subjective  workload  measures,  and  to  evaluate  the  usability  of  the  user  interface  and 
various  camera  feeds  (see  section  2. 4. 1.4  Eyetracker  Data). 


2.  Method 


2.1  Participants 

Thirty  individuals  (21  males,  9  females,  Mage  =  24.7  years)  from  the  Orlando,  FL,  area 
participated  in  the  experiment.  Participants  were  compensated  $  15/hr  for  their  time. 

2.2  Apparatus 
2.2.1  Simulator 

A  modified  version  of  the  Mixed  Initiative  Experimental  (MIX)  Testbed  was  used  as  the 
simulator  for  this  experiment  (Barber  et  ah,  2008).  The  RoboLeader  algorithm  was  implemented 
on  the  MIX  Testbed  and  it  had  the  capability  of  collecting  infonnation  from  subordinate  robots 
with  limited  autonomy  (e.g.,  collision  avoidance  and  self-guidance  to  reach  target  locations), 
making  tactical  decisions,  and  coordinating  the  robots  by  issuing  commands,  mapping 
waypoints,  etc.  (Snyder  et  ah,  2010).  The  MIX  Testbed  is  a  distributed  simulation  environment 
for  investigating  how  unmanned  systems  are  used  and  how  automation  affects  human  operator 
performance.  The  Operator  Control  Unit  (OCU)  of  the  MIX  Testbed  (figure  1)  was  modeled 
after  the  Tactical  Control  Unit  developed  under  the  ARL  Robotics  Collaborative  Technology 
Alliance. 
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Figure  1.  Operator  control  unit:  user  interface  for  convoy 
management  and  360°  tasking  environment. 


2.2.2  Eyetracker 

A  Sensomotoric  Instrument  (SMI)  Remote  Eyetracking  Device  (RED)  was  used  to  collect  eye 
movement  data.  The  SMI  RED  system  uses  an  infrared  (IR)  camera-based  tracking  system  and 
allows  completely  noncontact  operation  (SMI,  2013).  Eye  and  head  movements,  which  were 
observed  at  approximately  0.03°  of  spatial  resolution  and  sampled  at  the  rate  of  60  Hz,  along 
with  measurement  reliability  data,  were  logged  in  real  time  and  synchronized  with  performance 
data  from  the  MIX  Testbed  system.  Only  the  participants’  eye  gaze  coordinates  were  measured 
and  recorded,  no  video  of  the  participants’  faces  was  recorded.  The  SMI  system  was  individually 
calibrated  for  each  participant  prior  to  each  trial. 

2.2.3  RoboLeader 

RoboLeader  was  developed  to  assist  a  human  operator  manage  a  team  of  robots  (Chen  and 
Barnes,  2012a,  2012b;  Snyder  et  ah,  2010).  RoboLeader  is  a  mixed-initiative  system;  some 
processes  are  managed  by  RoboLeader  without  human  interaction  and  some  processes  offer 
suggestions  to  the  human  operator  for  approval  before  being  carried  out.  In  the  current  study, 
there  were  two  levels  of  assistance  from  RoboLeader:  Fully  Autonomous  and  Semi- 
Autonomous.  In  the  Fully  Autonomous  condition,  RoboLeader  managed  the  spacing  of  the 
convoy  vehicles  without  intervention  from  the  human  operator,  suggested  route  changes  when 
environmental  events  dictated,  and  then  made  the  route  changes  when  approved  by  the  operator. 
In  the  Semi-Autonomous  condition,  RoboLeader  managed  the  spacing  of  the  convoy,  but 
otherwise  offered  no  assistance  with  convoy  management  tasks. 

2.3.4  Survey  and  Tests 
2.3.4.1  Demographics  Survey 

A  demographics  questionnaire  was  administered  at  the  beginning  of  the  training  session. 
Information  on  participants’  age,  gender,  educational  level,  computer  familiarity,  and  gaming 
experience  was  collected  (appendix  A).  Participants  indicated  how  often  they  played  computer 
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or  video  games  (daily,  weekly,  monthly,  every  few  months,  rarely,  never),  and  participants  that 
responded  “daily”  or  “weekly”  were  categorized  as  frequent  players,  while  all  others  were 
infrequent.  Participants  also  reported  which  types  of  computer  games  they  play  most  often,  and 
game  type  was  categorized  as  Action  or  NonAction.  Action  games  were  defined  as  games  where 
the  majority  of  challenges  are  physical  tests  of  skill,  requiring  good  hand-eye  coordination  and 
quick  response  times,  and  that  have  a  time  constraint  as  well,  so  that  the  player  does  not  have 
time  for  complex  strategy  development  (Adams,  2010).  Examples  of  game  types  that  were 
classified  as  action  are  first-person  shooters  (FPS),  fighting  games,  racing  games  and  real-time 
strategy  games.  Participants  that  were  both  Frequent  and  Action  Gamers  were  categorized  as 
Frequent  Action  Gamers  (N=  13),  all  other  combinations  of  responses  were  All  Others  (N  =  17). 

2.2.4.2  Color  Vision  Test 

An  Ishihara  Color  Vision  Test  (using  nine  test  plates)  was  administered  via  PowerPoint 
presentation.  Because  the  RoboFeader  OCU  employs  several  colors  to  display  the  plans  for  the 
robots,  nonnal  color  vision  was  required  to  interact  effectively  with  the  system.  All  participants 
had  satisfactory  color  vision. 

2.2.4.3  Attentional  Control  Survey 

Participants’  PAC  was  evaluated  using  the  Attentional  Control  survey  (Derryberry  and  Reed, 
2002;  appendix  B).  The  Attentional  Control  survey  consists  of  21  items  and  measures  attention 
focus  and  shifting.  The  scale  has  been  shown  to  have  good  internal  reliability  (a  =  0.88). 
High/Fow  group  membership  was  determined  by  median  split  of  all  participants’  scores 
(PAClow/V  =  15,  PAChigh  N  =  15). 

2.2.4.4  Spatial  Ability  Tests 

The  Cube  Comparison  Test  (SpAC)  (Ekstrom  et  ah,  1976)  and  the  Spatial  Orientation  Test 
(SpAO)  (Gugerty  and  Brooks,  2004)  were  used  to  assess  participants’  spatial  ability.  These  two 
tests  measure  different  aspects  of  spatial  ability.  The  Cube  Comparison  Test  (appendix  C) 
estimates  the  ability  of  an  individual  to  perceive  spatial  patterns  and  to  maintain  orientation  with 
respect  to  objects  in  space.  It  requires  participants  to  compare,  in  3  min,  21  pairs  of  six-sided 
cubes  and  determine  if  the  rotated  cubes  are  the  same  or  different.  The  score  is  determined  by 
subtracting  the  number  of  incorrect  responses  from  the  number  of  correct  responses,  with  higher 
scores  indicating  higher  performance.  High/Fow  group  membership  was  determined  by  median 
split  of  all  scores  (SpACunv  N=  13,  SpACmoH  N  =  17). 

The  SpAO  (appendix  D),  modeled  after  the  cardinal  direction  test  developed  by  Gugerty  and  his 
colleagues  (Gugerty  and  Brooks,  2004),  evaluates  an  individual’s  ability  to  reorient  their 
imagined  self.  It  is  a  computerized  test  consisting  of  a  brief  training  segment  and  32  test 
questions.  The  program  automatically  captures  both  accuracy  and  response  time. 
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Individual  performance  scores  were  calculated  by  dividing  average  response  time  by  total 
number  correct,  with  higher  performance  indicated  by  lower  scores.  High/Low  group 
membership  was  determined  by  median  split  of  all  scores  (SpAOLow  N  =  15,  SpAOHiGH  N  =  15). 

1.2.4.5  NASA-TLX 

The  NASA-TLX  (appendix  E)  is  a  self-report  measure  of  perceived  workload.  The  computerized 
version  of  the  NASA-TLX  questionnaire  was  administered  after  each  trial  to  evaluate  perceived 
workload  (Hart  and  Staveland,  1988).  Definitions  of  each  subscale  were  provided  to  the 
participants  as  a  document  that  they  could  refer  to  as  needed  while  completing  the  questionnaire. 

2.2.4.6  Usability  and  Trust  Survey 

Participants’  perceived  usability  of  RoboLeader  and  overall  trust  in  the  system  were  evaluated 
using  the  Usability  and  Trust  Survey  from  Chen  and  Barnes  (2012a).  Trust  questions  were 
modified  from  Jian  et  al  (2000)  Trust  Between  People  and  Automation  questionnaire.  The  usability 
and  trust  survey  consisted  of  22  questions  rated  on  a  scale  of  1  to  7  (appendix  F).  Participants  were 
instructed  that,  for  the  purpose  of  the  survey,  “RoboLeader”  referred  to  the  Fully  Autonomous 
condition,  while  “RoboLeader  display”  referred  to  the  OCU  as  a  whole. 

2.3  Procedure 

After  being  briefed  on  the  purpose  of  the  study  and  signing  the  informed  consent  fonn, 
participants  completed  the  demographics  questionnaire,  the  Attentional  Control  Survey,  received 
a  brief  Ishihara  Color  Vision  Test,  and  completed  the  two  spatial  ability  tests. 

Participants  then  received  training  and  practice  on  the  tasks  needed  to  complete  the  experimental 
session.  Training  was  self-paced  and  delivered  via  PowerPoint*  slides.  The  slides  explained  the 
elements  of  the  OCU,  steps  for  completing  various  tasks,  and  included  several  practice  exercises 
for  performing  the  robotic  control  tasks.  The  training  session  lasted  approximately  1  h.  Before 
proceeding  to  the  experimental  session,  participants  had  to  demonstrate  that  they  could  perform 
the  tasks  without  any  help.  All  participants  demonstrated  adequate  mastery  of  the  system  before 
proceeding  with  the  study. 

The  experimental  session  lasted  approximately  1.5  h  and  began  immediately  after  the  training 
session.  Each  experimental  session  had  three  scenarios  (missions  1,  2,  and  3),  which  were 
presented  in  this  order  for  all  participants.  Using  a  Fatin  Square  design,  each  scenario  was 
randomly  assigned  an  automation  condition  (i.e.,  Manual,  Semi-Autonomous,  or  Fully 
Autonomous),  so  that  each  participant  completed  all  three  scenarios  and  experienced  all  three 
automation  levels.  Automation  condition  assignment  to  scenario  was  counter-balanced  to  avoid 
order  effects.  Scenarios  were  self-paced,  but  were  designed  to  take  25-40  min  to  complete. 

There  was  a  2  min  break  between  the  experimental  scenarios. 


PowerPoint  is  a  trademark  of  Microsoft  Corp. 
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During  the  scenarios,  participants  supervised  a  convoy  of  three  vehicles  (i.e.,  a  UAS,  a  UGV, 
and  their  MGV)  moving  from  point  A  to  point  B,  while  maintaining  specified  distances  between 
the  three  vehicles.  Each  scenario  had  a  route  mapped  for  the  vehicles  when  the  scenario  began, 
and  the  participants’  task  was  to  modify  the  routes  as  needed  based  on  environmental  or 
intelligence  “events.”  Several  events  occurred  during  each  scenario  that  required  revision  to  the 
convoy's  route.  These  events  could  be  in  the  form  of  intelligence  reports  that  the  human  operator 
received  from  the  intelligence  network  or  environmental  hazards  such  as  fire  or  road  blockages 
they  would  encounter  en  route.  Once  an  event  transpired,  the  participants  had  to  notice  and 
acknowledge  that  the  event  occurred,  and  then  respond  accordingly.  In  the  Fully  Autonomous 
condition,  RoboLeader  recommended  plan  revisions  for  the  events  by  presenting  the  new 
waypoints  on  the  map,  and  the  operator  could  accept  the  recommendation,  or  reject  and  modify 
the  route,  as  they  deemed  necessary.  In  the  Semi-Autonomous  and  Manual  conditions,  the 
participants  manually  modified  the  waypoints  for  the  lead  ground  vehicle  (i.e.,  UGV)  when  the 
convoy’s  route  needed  to  be  changed. 

Participants  were  advised  of  optimal  spacing  requirements  for  between  vehicle  distances,  and 
part  of  their  convoy  management  duties  were  to  supervise  and  maintain  these  required  distances. 
Vehicle  separation  was  accomplished  by  pausing  and  restarting  individual  vehicle  movement, 
which  allowed  trailing  vehicles  to  catch  up  to  lead  vehicles  or  allowed  lead  vehicles  to  increase 
distance  from  trailing  vehicles.  In  the  RoboLeader  conditions  (Semi-Autonomous  and  Fully 
Autonomous),  the  distance  separations  among  the  vehicles  were  maintained  automatically  based 
on  the  vehicles’  own  leader- follower  algorithms. 

In  addition  to  the  convoy  management  duties,  participants  had  to  maintain  360°  local  security 
surrounding  their  own  MGV  by  simultaneously  monitoring  the  UGV  forward  camera  feed  and 
the  MGV’s  two  180°  indirect-vision  displays  (one  forward  view,  one  rearward  view)  to  detect 
hostile  targets  in  the  immediate  environment.  Once  a  hostile  target  was  detected,  the  participants 
would  “laze”  the  target  by  clicking  on  the  target  using  the  mouse.  An  icon  representing  the 
“lazed”  insurgent  would  then  be  displayed  on  the  map.  Hostile  targets  were  defined  as  armed 
civilian  individuals  or  vehicles.  Friendly  dismounted  soldiers  and  unarmed  civilians  were  present 
in  the  simulated  environment  to  increase  the  visual  noise  present  in  the  target  detection  tasks. 
Detection  difficulty  was  balanced  within  each  scenario,  with  some  targets  only  visible  upon 
approach,  some  only  visible  in  the  rear  camera  feed,  and  some  visible  both  approaching  and  after 
passing. 

Each  scenario  contained  five  SA  queries  (appendix  G),  which  were  triggered  based  on  time 
progression  (i.e.,  3  min  into  the  scenario,  and  then  at  4  min  intervals  for  the  remainder  of  the 
trial).  When  an  SA  query  appeared,  the  OCU  screen  went  blank,  the  simulation  paused,  and  the 
SA  query  was  displayed  on  the  screen.  Participants  then  wrote  their  response  to  the  query  on  the 
answer  sheet  for  that  scenario  (appendix  H).  After  participants  responded  to  the  SA  query,  they 
clicked  the  “Continue”  button  on  the  screen  for  the  SA  screen  to  disappear,  and  the  simulation 
resumed. 
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Participants  assessed  their  perceived  workload  immediately  after  each  experimental  scenario 
using  the  computerized  version  of  the  NASA-TLX.  Following  completion  of  all  three  scenarios, 
participants  evaluated  the  usability  of  the  RoboLeader  system  by  completing  the  Usability/Trust 
Questionnaire  (Jian  et  ah,  2000).  Participants  were  then  debriefed  by  the  experimenter  and 
dismissed. 

2.4  Experimental  Design  and  Performance  Measures 

This  study  was  a  mixed  within-hetween  subject  design,  with  Level  of  Autonomy  (Manual  [no 
RoboLeader],  Semi-Autonomous,  and  Fully  Autonomous)  as  the  within-subject  variable  and 
individual  difference  factors  such  as  spatial  ability,  attentional  control,  and  gaming  experience  as 
the  between-subjects  variables.  Dependent  measures  were  the  numbers  of  targets  identified,  false 
alarms  (FAs),  the  operators’  SA  of  the  mission  environment,  and  perceived  workload. 

2.4.1  Dependent  Measures 

2.4.1. 1  Target  Detection 

Target  detection  perfonnance  has  been  shown  to  be  related  to  spatial  ability  (Chen  and  Joyner, 
2009;  Chen  and  Barnes,  2012a),  possibly  due  to  improved  scanning  abilities  (Thomas  and 
Wickens,  2004).  Each  scenario  contained  30  targets;  however,  participants  were  only  scored  on 
targets  that  were  in  close  proximity  to  their  chosen  route.  Target  detection  performance  was 
calculated  by  dividing  the  number  of  targets  correctly  identified  by  the  total  number  of  targets 
that  were  available  to  the  participant  for  detection.  FAs  were  also  examined  as  a  measure  of 
interest,  as  the  number  of  FAs  should  increase  as  cognitive  workload  increases.  Each  scenario 
had  several  hundred  potential  distractors  (noise);  however,  there  were  very  few  reported  FAs 
across  all  conditions  and  scenarios,  which  would  result  in  FA  Rates  so  small  as  to  be 
indistinguishable  from  one  another  using  the  Signal  Detection  Theory  formula.  Because  of  this, 
the  actual  number  of  reported  FAs  is  used  in  all  calculations. 

2.4.1.2  Situation  Awareness 

Situation  awareness  is  the  perception  and  comprehension  of  the  elements  within  one’s 
environment  and  the  projection  of  their  future  states  (Endsley,  1995),  and  is  therefore  essential  to 
appropriate  decision-making  and  supervisory  control  (Chen  et  ah,  2011).  Multitasking  divides  an 
individual’s  attention,  making  it  difficult  to  maintain  appropriate  levels  of  SA  when  switching 
between  tasks  (Cummings,  2004).  Chen  and  Barnes  (2012a)  found  participants  with  higher  SpA 
and  experienced  gamers  maintained  higher  SA  than  those  with  low  SpA  and  infrequent  gamers. 

2.4.1.3  Workload 

Vigilance  tasks,  such  as  maintaining  local  security  (target  detection  tasks),  are  particularly 
demanding  of  cognitive  resources.  As  resources  become  limited,  attentional  effort  is  reduced  and 
mental  workload  increases  (Warm  et  ah,  1996;  Warm  et  ah,  2008).  Targets  that  are  less  salient  or 
randomly  placed  in  the  environment  further  increase  the  demands  of  the  task  on  the  operator, 
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increasing  workload  even  more  (Warm  et  al.,  1996).  Increased  workload  causes  the  operator  to 
rely  on  the  automation  more,  resulting  in  an  operator  “out  of  the  loop”  situation.  This  causes  the 
operators  to  miss  errors  and  rely  on  the  system  to  conduct  tasks  with  little  or  no  supervision 
(Ruff  et  al.,  2002).  Higher  levels  of  automation  typically  reduce  operator  workload,  resulting  in 
improved  SA  and  change  detection  accuracy  as  long  as  the  operator  is  engaged  in  the  tasks  that 
are  automated  (Parasuraman  et  al.,  2009).  Persons  with  high  PAC  have  been  found  to  be  more 
resilient  in  multitasking  situations  than  those  with  low  PAC,  performing  better  even  during  more 
challenging  situations  and  on  secondary  tasks  (Chen  and  Barnes,  2012a;  Chen  and  Joyner,  2009). 

2.4.1.4  Eyetracker  Data 

2.4. 1.4.1  Glances:  Number  and  Duration 

A  glance  (or  dwell)  is  defined  as  one  visual  visit  in  a  predefined  area,  from  entry  to  exit 
(Hohnqvist  et  al.,  2011).  The  number  of  glances  in  a  predefined  area  is  positively  correlated  to 
instrument  difficulty  (Chisholm  et  al.,  2008),  as  well  as  semantic  informativeness  (Loftus  and 
Mackworth,  1978). 

Glance  duration  (aka  dwell  time)  is  defined  for  purposes  of  this  study  as  beginning  at  the  time 
the  area  of  interest  (AOI)  is  first  fixated  on  by  an  individual  until  the  end  of  the  last  fixation  in 
the  AOI  for  one  visit.  This  includes  the  summation  of  all  fixations  and  saccades  that  occur  during 
each  glance.  Glance  duration  is  one  of  the  most  widely  used  measures  in  usability  studies  (Jacob 
and  Kam,  2003),  most  likely  due  to  its  versatility.  Glance  duration  has  been  found  to  be 
indicative  of  several  negative  perfonnance  situations,  such  as  difficulties  extracting  information 
(Goldberg  and  Kotval,  1999),  poor  SA  (Hauland,  2008),  and  task  uncertainty  (Ottati  et  al.,  1999). 
However,  it  can  also  indicate  higher  interest,  the  infonnativeness  of  an  object  (Friedman  and 
Liebelt,  1981),  and  when  a  conscious  choice  is  impending  (Shimojo  et  al.,  2003). 

2.4. 1.4.2  Fixations 

Fixations  are  low-velocity  eye  movements  that  correspond  to  a  person  staring  at  a  particular 
point.  They  contain  small  randomly  drifting  eye  movements  and  quick  adjustments  to  keep  a 
particular  target  centered.  The  SMI  eye  tracking  system  detects  fixations  by  applying  a 
maximum-movement  threshold  amount  for  a  minimum  period  of  time  (SMI,  2011).  The  number 
of  fixations  is  also  frequently  seen  in  usability  studies  and  is  positively  correlated  with  glance 
duration  (Jacob  and  Karn,  2003).  The  number  of  fixations  in  a  predefined  area  has  been  found  to 
be  positively  correlated  to  semantic  importance  (Jacob  and  Kam,  2003),  search  difficulty 
(Ehmke  and  Wilson,  2007),  and  encoding  memory  (i.e.,  more  fixations  on  an  object  result  in 
better  memory  for  that  object;  Tatler  et  al.,  2005).  Fixations  are  negatively  correlated  with  search 
efficiency  (Goldberg  and  Kotval,  1999),  experience  (Reingold  et  al.,  2001)  and  increased  mental 
workload  (Van  Orden  et  al.,  2000).  Studies  on  expertise  have  indicated  that  experts  make  more 
fixations  of  shorter  duration  than  novices,  demonstrating  improved  search  efficiency  (Kasarskis 
et  al.,  2001;  Williams  et  al.,  1994). 
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2. 4. 1.4. 3  Blink  Rate 


Blink  rate  is  defined  as  the  number  of  blinks  per  second.  Blink  rate  tends  to  increase  with  time 
on  task,  fatigue,  time  of  day  (Stern  et  ah,  1994),  and  increased  mental  workload  (Van  Orden  et 
al„  2000). 

2.4. 1.4.4  Saccade  Amplitude 

The  amplitude  of  a  saccade  is  the  distance  travelled  by  a  saccade  from  its  onset  to  its  end.  It  can 
be  measured  either  in  visual  degrees  or  in  pixels  (Holmqvist  et  ah,  2011).  Saccade  amplitude 
tends  to  be  idiosyncratic,  and  as  such,  it  was  used  only  as  a  repeated  measures  measure  in  this 
study.  Shorter  saccadic  amplitudes  are  correlated  with  difficulty  in  search  tasks  (Zelinsky  and 
Sheinberg,  1997),  and  increased  mental  workload  (May  et  ah,  1990;  Recarte  and  Nunes,  2003). 
They  are  also  indicative  of  density  of  visual  information  in  the  area  surrounding  fixation  (Tatler 
et  ah,  2006). 

2.4. 1.4.5  Pupil  Diameter 

Pupil  size  is  raw  data  that  is  sampled  throughout  the  recording  of  the  eye  tracking  system.  It  is 
sensitive  to  lighting  changes  in  the  stimulus,  view  angles  (i.e.,  gaze  direction),  distance  to  the 
screen  and  eyelid  closure.  Pupil  size  is  measured  by  imposing  an  ellipse  over  the  dark  area  of  the 
pupil  and  reporting  the  vertical  and  horizontal  axis  lengths;  this  is  sometimes  reported  as  vertical 
and  horizontal  diameters.  The  horizontal  axis  measurement  is  less  sensitive  to  artifacts  due  to 
eyelid  closure  than  the  vertical  axis  measurement  (Holmqvist  et  ah,  2011).  Increase  in  pupil 
diameter  has  been  found  to  be  positively  correlated  with  increased  mental  workload  (Van  Orden 
et  ah,  2000;  Van  Orden  et  ah,  2001;  Iqbal  et  ah,  2004),  and  heightened  interest  (Kang  et  ah, 
2009). 

2.4.2  Operator  Control  Unit  and  Areas  of  Interest 

The  OCU  screen  was  divided  into  14  AOIs  (figure  2).  AOIs  were  defined  primarily  by  their 
function,  and  some  AOIs  were  then  subdivided  based  on  screen  position  and  size.  The  central 
AOIs  were  expected  to  have  the  most  fixations  and  dwells,  as  orienting  the  eye  on  a  central 
portion  of  the  screen  appears  to  allow  for  better  early  information  intake  and  responsiveness 
(Tatler,  2007;  Fehd  and  Seiffert,  2010).  The  number  of  fixations,  number  of  glances,  and  average 
glance  duration  for  each  AOI  were  gathered  and  analyzed,  together  with  target  detection 
performance  and  workload  ratings  to  investigate  utility  of  AOI  content  and  placement. 
Participants’  scanning  behavior  and  efficiency  was  also  examined. 
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Figure  2.  OCU  screen  divided  into  areas  of  interest. 


3.  Results 


Repeated  measures  ANOVAs  were  used  to  evaluate  the  effect  of  LOA  on  the  dependent 
variables,  a  =  .05.  Mixed  within-between  subjects  ANOVAs  were  used  to  evaluate  the  influence 
of  individual  difference  factors  on  the  dependent  measures  across  three  LOA,  a  =  .05.  Findings 
were  reported  as  significant  when p  <  .050,  marginally  significant  when p  <  .075.  Pairwise 
comparisons  were  then  used  to  explore  specific  differences  across  condition  levels  when 
significant  results  were  found. 

The  effect  of  LOA  on  eye  behavior  measures  was  evaluated  using  within-subjects  ANOVAs. 
Relationships  between  eye  behavior  measures  and  DVs  (SA,  Workload,  and  Usability)  were 
examined  using  correlations.  Mixed  within-between  subjects  ANOVAs  were  used  to  evaluate  the 
influence  of  individual  difference  factors  on  eye  behavior  measures  (Fixation  Count,  Average 
Fixation  Duration,  and  Pupil  Diameter)  across  three  LOA,  a  =  0.05.  No  between- subjects 
evaluations  were  performed  on  Blink  Rate  or  Saccade  Amplitude,  as  these  measures  tend  to  be 
idiosyncratic.  Pairwise  comparisons  were  then  used  to  explore  specific  differences  across 
condition  levels  when  significant  results  were  found.  When  2-tailed  results  are  reported,  p  < 
0.050  is  reported  as  significant,/!  <  0.090  is  moderately  significant. 

The  Chi-Square  Test  for  Independence  was  used  to  evaluate  the  relationship  between  the 
independent  variables  (Gaming  Freq/ Action,  PAC,  SpAC  and  SpAO).  The  procedure  used  a  2  x 
2  design,  and  the  Yates’  Correction  for  Continuity  was  used  to  control  for  over-estimation  of  the 
2x2  table.  All  IVs  were  found  to  be  independent  of  one  another. 

Participant  6  had  a  missing  score  for  NASA-TLX  (Semi-Autonomous  condition);  this  was 
replaced  with  the  average  of  their  scores  for  the  Fully  Autonomous  and  Manual  missions.  As 
evidenced  by  other  cases,  the  semi-autonomous  score  typically  falls  midway  between  the  other 
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two  scores,  so  averaging  them  appears  to  be  an  appropriate  method  to  allow  us  to  keep  the  data 
for  this  participant. 

Several  DVs  (i.e.,  FAs,  Total  Pause  Time,  Total  Simulation  Time,  and  SA)  had  extreme  outliers 
that  resulted  in  skewed  or  kurtotic  distributions.  According  to  Tabachnick  and  Fidell  (2007,  pg. 
77),  an  acceptable  method  to  reduce  the  influence  of  univariate  outliers  is  to  simply  replace  the 
extreme  value  with  a  value  that  is  one  unit  larger  (or  smaller)  than  the  next  most  extreme  score, 
thus  reducing  their  influence  while  maintaining  their  relation  to  the  rest  of  the  data.  Univariate 
outliers  were  handled  in  this  manner  for  all  perfonnance  DVs.  Although  participant  33 ’s  data  is 
not  technically  a  univariate  outlier  (as  their  scores  were  extreme  outliers  on  two  measures  (Fully 
Autonomous  FAs  and  Manual  SA),  the  two  measures  are  not  directly  related  and  adjusting  those 
values  using  the  univariate  method  should  not  have  a  detrimental  effect  on  the  whole. 

When  the  eyetracker  data  was  evaluated  for  assumptions,  it  was  found  that  multiple  steps  were 
often  required  to  sufficiently  clean  the  data.  When  extreme  outliers  (>3  SD)  were  encountered  in 
the  first  analysis,  they  were  removed  and  the  data  re-assessed.  Any  outliers  identified  after  the 
initial  cleaning  were  considered  for  correction  via  the  Tabachnick  and  Fidell  (2007)  method.  No 
further  data  transfonnations  or  manipulations  were  required. 

3,1  Performance  Measure  Findings 

Overall  main  effects  are  summarized  in  table  1 . 


Table  1.  Mean  operator  task  performance  and  workload  assessments. 


Measure 

Manual 

Semi 

Full 

Main  Effects 

Target  Detection 

%  detected  (SD) 

54.6(13.3) 

65.3  (14.1) 

63.5  (12.0) 

Manual  <  Semi  (p  <  0.001) 
Manual  <  Full  ( p  =  0.001) 

False  Alarms 

#  Reported  (SD) 

2.43  (1.85) 

3.23  (2.10) 

2.40  (1.65) 

Manual  <  Semi  (p  =  0.058) 
Semi  >  Full  (p  =  0.047) 

Situation  Awareness 

%  Correct  (SD) 

70.7(3.9) 

76.0(3.4) 

76.7(3.7) 

— 

Workload 

NASA-TLX  Score  (SD) 

69.09  (14.13) 

55.24  (11.54) 

47.49  (19.21) 

Manual  >  Semi  (p  <  0.001) 
Manual  >  Full  (p  <  0.001) 
Semi  >  Full  (p  <  0.001) 

3.1.1  Target  Detection  Performance 

3.1.1. 1  LOA 

There  was  a  significant  difference  in  Target  Detection  Performance  across  three  LOAs,  Wilks’  X 
=  0.583,  F  (2,  28)  =  10.007,  p  =  0.001,  partial  if  =  0.417.  Participants  detected  significantly 
fewer  targets  in  the  Manual  condition  ( M=  54.5%),  than  in  the  Semi-Autonomous  condition  (M 
=  65.3%,  AM=  10.8 %,p<  0.001),  or  in  the  Fully  Autonomous  condition 
(M=  63.5%,  AM=  8.9%, p  =  0.001),  figure  3. 
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Figure  3.  Target  detection  performance  across 
LOAs. 

3.1. 1.2  SpAC 

There  was  not  a  significant  interaction  between  SpAC  and  LOA  on  Target  Detection 
Performance,  Wilks’  X  =  0.991,  F  (2,  27)  =  0.128 ,p  =  0.881,  partial  rj2  =  0.009.  There  was  no 
significant  between-subjects  effect  for  SpAC  on  Target  Detection,  F  (1,  28)  =  0.284,  p  =0.598, 
partial  rf  =  0.010. 

3. 1.1.3  SpAO 

There  was  a  significant  interaction  between  SpAO  and  LOA  on  Target  Detection  Performance, 
Wilks’  X  =  0.770,  F  (2,  27)  =  4.025, p  =  0.030,  partial  rj 2  =  0.230  (figure  4).  All  participants 
benefitted  from  automation  assistance  in  the  RoboLeader  conditions.  Participants  with  high 
SpAO  scores  showed  greatest  improvement  in  the  Semi-Autonomous  condition,  with  their  scores 
in  the  Fully  Autonomous  condition  being  similar  to  those  with  low  SpAO  scores.  Each 
increasing  LOA  helped  the  low  SpAO  participants  improve  their  performance  on  the  target 
detection  task,  with  performance  in  the  Fully  Autonomous  condition  being  similar  to  those  with 
high  SpAO. 


Figure  4.  Target  detection  performance  by  SpAO, 
across  LOAs. 
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3.1.2  False  Alarms 


3. 1.2.1  LOA 

There  was  no  significant  difference  in  reported  FAs  due  to  LOA,  Wilks’  X  =  0.834,  F  (2,  28) 

=  2.786,  p  =  0.079,  partial  //  =  0.166.  Participants  reported  more  FAs  in  the  Semi-Autonomous 
condition  (M=  3.233)  than  in  either  Manual  (M=  2.433,  AM=  -0.800,/?  =  0.058)  or  Fully 
Autonomous  (M=  2.400,  AM  =  -0.833,/?  =  0.047);  however,  the  overall  ANOVA  differences 
did  not  reach  significance  (figure  5). 


Figure  5.  Mean  FAs  by  LOA. 


3.1.3  Situation  Awareness 

3.1.3. 1  LOA 

There  was  no  significant  difference  in  SA  scores  due  to  LOAs,  Wilks’  X  =  0.941,  F  (2,  28) 

=  0.879,/?  =  0.427,  partial  if  =  0.059. 

3. 1.3.2  SpAC 

There  was  not  a  significant  interaction  between  SpAC  and  LOA  on  SA,  Wilks’  X  =  0.971,  F  (2, 
27)  =  0.403,/?  =  0.672,  partial  rj  =  0.029.  There  was  no  significant  between-subjects  effect  for 
SpAC  on  SA  scores,  F(l,  28)  =  0.001,/?  =  0.971,  partial  rj2  =  0.000. 

3. 1.3.3  SpAO 

There  was  a  significant  interaction  between  SpAO  and  LOA  on  SA,  Wilks’  X  =  0.767,  F  (2,  27) 
=  4.103,/?  =  0. 028,  partial  rj  =  0.233  (figure  6).  Participants  with  high  SpAO  maintained  higher 
SA  across  LOA  than  those  with  lower  SpAO,  and  scored  higher  on  the  SA  task  during  the 
Manual  condition  (84.0%  correct)  than  those  with  lower  SpAO  (57.4%  correct).  Although 
RoboLeader  helped  those  with  low  SpAO  perform  better  on  the  SA  task  in  both  Semi-  and  Fully 
Autonomous  conditions;  it  did  not  appear  to  benefit  those  with  high  SpAO. 
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Figure  6.  SA  scores  by  SpAO,  across  LOAs. 

3.1.3.4  Gaming  Experience 

There  was  no  significant  interaction  between  Gaming  Experience  and  LOA  on  SA,  Wilks’  X  = 
0.842,  F  (2,  27)  =  2.536,  p  =  0.098,  partial  rj2  =  0.158.  There  was  a  significant  between-subjects 
effect  for  Gaming  Experience  on  SA,  F  (1,  28)  =  4.523,  p  =  0.042,  partial  rj2  =  0.139,  with 
Frequent  Action  Gamers  scoring  significantly  higher  on  SA  measures  in  the  Manual  and  Fully 
Autonomous  conditions  (figure  7). 
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Figure  7.  SA  scores  by  gaming  experience,  across 
LOAs. 


3.1.4  Perceived  Workload 
3.1.4.1  LOA 

There  was  a  significant  difference  in  Perceived  Workload  across  three  LOAs.  Participants 
reported  a  significantly  higher  level  of  perceived  workload  in  the  Manual  condition  (M=  69.09) 
than  in  the  Semi-Autonomous  (M=  55.24,  AM=  13.850,/?  <  0.001)  and  Fully  Autonomous 
(M=  47.49,  AM=  21.60,/?  <  0.001)  conditions,  and  a  significantly  higher  level  in  the  Semi- 
Autonomous  condition  than  in  the  Fully  Autonomous  condition  ( AM  =  7.75,  p  =  0.004),  Wilks’ 
X  =  0.354,  F  (2,  28)  =  25.500,/?  <  0.001,  partial  rj2  =  0.646  (figure  8). 
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Figure  8.  Perceived  workload  (NASA-TLX 
scores)  across  LOAs. 


3. 1.4.2  PAC 

There  was  no  significant  interaction  between  PAC  and  LOA  on  Perceived  Workload,  Wilks’ 

X  =  0.989,  F  (2,  27)  =  0.151  ,p  =  0.860,  partial  rj2  =  0.01 1.  There  was  no  significant  between- 
subjects  effect  for  PAC  on  Perceived  Workload,  F  (1,  28)  =  0.082,/?  =  0.777,  partial  rj2  =  0.003. 

3.2  Eyetracker  Findings 

3.2.1  Fixations:  Count  and  Duration 

3.2. 1.1  FOA 

Fixation  Count  was  significantly  lower  for  Manual  condition  than  for  Semi-  and  Fully 
Autonomous  conditions,  Wilks’  X  =  0.578,  F  (2,  24)  =  8.775,  p  =  0.001,  partial  rj2  =  0.422  (figure 
9a).  Number  of  Fixations  was  significantly  lower  in  the  Manual  condition  (M=  4384)  than  in  the 
Semi- Autonomous  (M=  5067,  AM  =  682,  p  <  0.001)  or  Fully  Autonomous  (M=  5007,  AM  = 
623,  p  =  0.001)  conditions  (figure  9a).  Average  Fixation  Duration  was  significantly  higher  in  the 
Manual  condition  (M=  246.90)  than  in  the  Fully  Autonomous  condition,  M=  234.63,  AM=  12.27, 
p  <  0.05  (figure  9b). 


Figure  9.  Fixation  count  (a)  and  average  fixation  duration  (b)  by  LOA. 
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3.2. 1.2  Perceived  Workload  (NASA-TLX) 

There  was  a  significant  correlation  between  Fixation  Count  and  Perceived  Workload  for  the 
Manual  condition,  r  =  0.470,/)  =  0.009,  2-tailed  (table  2);  however,  Fixation  Count  did  not 
correlate  with  Perceived  Workload  in  either  of  the  RoboLeader  conditions,  nor  did  Fixation 
Duration  correlate  with  Workload  in  any  condition. 

3.2. 1.3  SA 

Fixation  Count  was  negatively  correlated  with  SA  score  for  the  Semi-Autonomous  condition, 
r  =  -0.435,/)  =  0.021,  2-tailed  (table  1),  but  had  a  moderate,  positive  correlation  for  Manual 
(r  =  0.31  \,p  =  0.095,  2-tailed)  and  Fully  Autonomous  (r  =  0.331,/)  =  0.085,  2-tailed)  conditions. 
Average  Fixation  Duration  did  not  correlate  with  SA  scores  in  any  condition. 

3.2. 1.4  Usability  Rating 

Fixation  Count  and  Average  Fixation  Duration  did  not  correlate  with  Usability  ratings  in  any 
condition. 

Table  2.  Correlations  for  fixation  count  and  fixation  duration,  and  performance  measures. 


Fixation  Count 

Fixation  Duration 

Manual 

Semi 

Full 

Manual 

Semi 

Full 

r 

Sig. 

r 

Sig. 

r 

Sig. 

r 

Sig. 

r 

Sig. 

r 

Sig. 

NASA-TLX 

0.470a 

0.009 

0.070 

0.722 

0.272 

0.162 

0.231 

0.220 

0.185 

0.329 

-0.006 

0.975 

SA  Scores 

0.311 

0.095 

-0.435b 

0.021 

0.331 

0.085 

0.239 

0.204 

0.119 

0.530 

-0.257 

0.170 

Usability 

-0.052 

0.785 

0.068 

0.732 

-0.123 

0.534 

-0.058 

0.759 

-0.005 

0.978 

0.102 

0.591 

Correlation  is  significant  at  the  0.01  level  (2-tailed) 
Correlation  is  significant  at  the  0.05  level  (2-tailed) 


3.2. 1.5  Individual  Difference  Factors 

There  were  no  significant  interactions  or  main  effects  of  SpAC,  SpAO,  or  Gaming  Experience  on 
Fixation  Count  or  Average  Fixation  Duration. 

3.2. 1.6  PAC 

There  was  no  significant  interaction  between  PAC  and  LOA  on  Fixation  Count,  Wilks’ 

X  =  0.958,  F  (2,  27)  =  0.498,  p  =  0.614,  partial  tf  =  0.042.  There  was  a  marginally  significant 
between-subjects  effect  for  PAC  on  Fixation  Count,  F  (1,  28)  =  3.747,/)  =  0.065,  partial  rj  = 
0.135,  (figure  10).  Participants  with  low  PAC  had  more  Fixations  than  those  with  high  PAC, 
across  all  LOAs. 
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Fixation  Count  by 
Attentional  Control 


Manual  Semi  Ful 


Figure  10.  Number  of  fixations  by  PAC, 
across  LOAs. 

There  was  not  a  significant  interaction  between  PAC  and  LOA  on  Average  Fixation  Duration, 
Wilks’  X  =  0.987,  F  ( 2,  27)  =  0.173 ,p  =  0.842,  partial  rj2  =  0.013.  There  was  no  significant 
between-subjects  effect  for  PAC  on  Average  Fixation  Duration,  F  (1,  28)  =  0.872,  p  =  0.358, 
partial  rj2  =  0.030. 

3.2.2  Blink  Rate 
3.2.2. 1  LOA 

There  was  no  significant  difference  in  Blink  Rate  across  three  LOAs,  Wilks’  X  =  0.841,  F  (2,  24) 
=  2.451,  p  =  0.106,  partial  rj2  =  0.159.  Blink  Rate  for  Manual  condition  (M=  14.20)  was  higher 
than  for  Semi-Autonomous  (M=  1 1.02,  AM=  3.183 ,p  =  0.041),  or  Fully  Autonomous  (M  = 
10.99,  AM  =  3.206, p  =  0.06 1)  (figure  1 1). 


Blink  Rate 


Manual  Semi  Full 


Figure  1 1 .  Blink  rate  by  LOA. 


3.2.2.2  Performance  Measures 

Blink  Rate  was  not  correlated  with  any  Performance  Measures  (table  3). 
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Table  3.  Blink  rate  correlations  with  performance  measures. 


Blink  Rate 

Manual 

Semi 

Full 

r 

Sig. 

r 

Sig. 

r 

Sig. 

NASA-TLX 

0.072 

0.706 

-0.033 

0.865 

-0.189 

0.317 

SA  Scores 

-0.146 

0.441 

0.113 

0.552 

-0.142 

0.455 

Usability 

0.089 

0.640 

-0.018 

0.926 

0.052 

0.784 

3.2.3  Saccade  Amplitude 
3.2.3. 1  LOA 

The  average  Saccadic  Amplitude  was  significantly  shorter  for  Semi-Autonomous  condition 
(M=  7.800)  than  Manual  (M=  10.633,  AM  =  2.833, p  =  0.041),  and  shorter  but  not  significantly 
so  for  Fully  Autonomous  (M=  1 1.500,  AM  =  3.700,/?  =  0.252),  Wilks’  X  =  0.809,  F  (2,  28)  = 
3.295,/?  =  0.052,  partial  rf  =  0.191  (figure  12). 
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Figure  12.  Average  saccade  amplitude  by 
LOA. 


3.2.3. 2  Perceived  Workload 

Perceived  workload  was  negatively  correlated  with  participants’  average  saccade  amplitude  in 
the  Manual  and  Semi-Autonomous  conditions,  but  was  not  significant  in  the  Fully  Autonomous 
condition  (table  4). 
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Table  4.  Saccade  amplitude  correlations  with  performance  measures. 


Saccade  Amplitude 

Manual 

Semi 

Full 

r 

Sig. 

r 

Sig. 

r 

Sig. 

NASA-TLX 

-0.425* 

0.019 

-0.342 

0.064 

-0.146 

0.441 

SA  Scores 

-0.310 

0.095 

0.108 

0.571 

-0.023 

0.904 

Usability 

0.140 

0.461 

0.158 

0.404 

0.220 

0.242 

3.2.4  Pupil  Diameter 
3.2.4.1  LOA 

The  average  Pupil  Diameter  for  Manual  condition  (M=  14.667)  was  significantly  larger  than  for 
Semi-Autonomous  (M=  14.133;  AM=  0.533,/?  =  0.007),  and  Fully  Autonomous  (M=  13.800; 
AM  =  0.867,  p  =  0.002);  Wilks’  X  =  0.687,  F  (2,  28)  =  6.372,  p  =  0.005,  partial 
rj2  =  0.313  (figure  13). 


Pupil  Diameter 

15.5  - 


Manual  Semi  Full 


Figure  13.  Average  pupil  diameter  by  LOA. 


3.2.4.2  Performance  Measures 

Pupil  Diameter  was  not  correlated  with  any  Performance  Measures  (table  5). 


Table  5.  Average  pupil  diameter  correlations  with  performance  measures. 


Average  Pupil  Diameter 

Manual 

Semi 

Full 

r 

Sig. 

r 

Sig. 

r 

Sig. 

NASA-TLX 

0.007 

0.969 

-0.190 

0.314 

-0.032 

0.866 

SA  Scores 

-0.208 

0.270 

0.289 

0.121 

-0.187 

0.322 

Usability 

0.010 

0.958 

-0.198 

0.293 

-0.170 

0.368 
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3 .2.4.3  PAC 


No  significant  interaction  was  found  between  PAC  and  LOA  on  Pupil  Diameter,  Wilks’  X  = 
0.847,  F  (2,  27)  =  2.445,/?  =  0.106,  partial  rj2  =  0.153,  ns. 

There  was  a  marginally  significant  effect  of  PAC  on  Average  Pupil  Diameter  across  three  LOAs, 
F(  1,  28)  =  3.899, p  =  0.058,  partial  rj2  =  0.122,  (figure  14).  Average  Pupil  Diameter  for 
participants  with  low  PAC  was  consistent  across  all  mission  conditions,  while  Average  Pupil 
Diameter  for  participants  with  high  PAC  was  greatest  in  the  Manual  condition,  smaller  in  the 
Semi-Autonomous  condition  and  smallest  in  the  Fully  Autonomous  condition,  indicating  higher 
mental  workload  in  the  lower  levels  of  automation  conditions. 


Figure  14.  Pupil  diameter  by  PAC,  across  LOAs. 


3.3  Operator  Control  Unit  Usability  Analysis 

3.3.1  AOI  Analysis 

Each  AOI  (see  figure  2)  was  evaluated  on  a  number  of  measures  relating  to  how  useful  the 
participant(s)  found  that  particular  AOI  to  be,  and  how  its  usage  related  to  overall  performance  in 
the  Target  Detection  Task  and  number  of  FAs  reported.  Additionally,  AOI  usage  was  examined 
for  differential  usage  based  on  individual  difference  factors.  General  findings  are  included  in  this 
section  (figure  15);  specific  results  by  AOI  are  reported  in  appendix  I. 

3.3. 1.1  Target  Detection  Performance 

Average  Fixation  Duration  in  AOI  8  (UGV  Camera  Feed)  was  predictive  of  performance  on  the 
Target  Detection  Task  in  the  Fully  Autonomous  condition;  however,  it  was  not  predictive  of  task 
performance  for  any  other  AOI  or  any  other  LOA  (appendix  I,  table  9). 
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LOA1-2-3-4 

PAC4 

SpAO4 

SpAC1-3 


Prcd  Targ  Det  Performance3-4 
Pred  l  As1-2 

LOA1-3-4  12  SpAC1-3 

PAC4  Clicking 


1.  Fixation  Count 

2.  Average  Fixation  Duration 

3.  Glance  Count 

4.  Average  Glance  Duration 


Figure  15.  AOI  significant  findings,  mapped  onto  OCU. 

3.3. 1.2  False  Alarms 

Fixation  Count,  Average  Fixation  Duration,  Glance  Count,  and  Average  Glance  Duration  in 
AOI  12  (Rearward  Right  Camera  Feed)  were  all  predictive  of  reported  False  Alarms  (appendix  I, 
table  30).  Although  more  Fixations  and  longer  duration  Fixations  resulted  in  fewer  reported 
False  Alarms,  more  Glances  and  longer  duration  Glances  resulted  in  more  reported  False 
Alarms. 

3.3. 1.3  LOA 

There  was  a  main  effect  of  LOA  on  Fixation  Count,  Average  Fixation  Duration,  Glance  Count, 
and  Average  Glance  Duration  in  AOIs  7  (UAS  Camera  Feed),  8  (UGV  Camera  Feed), 

9  (Forward  Right  Camera  Feed),  1 1  (Forward  Left  Camera  Feed),  12  (Rearward  Right  Camera 
Feed),  and  14  (Rearward  Left  Camera  Feed).  There  were  fewer  and  shorter  duration  Fixations 
and  Glances  in  these  AOIs  during  the  Manual  condition  than  in  either  of  the  RoboLeader 
conditions. 
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3.3. 1.4  Attention  Control 


High  PAC  participants  had  longer  duration  glances  in  AOI  9  and  12  than  low  PAC  participants 
in  all  LOAs.  Additionally,  there  was  an  interaction  between  PAC  and  LOA  for  AOI  14. 
Participants  with  high  PAC  had  fewer  Fixations  and  glances,  and  shorter  duration  glances,  than 
low  PAC  participants  in  the  Manual  condition,  but  there  was  no  difference  due  to  PAC  for  this 
AOI  during  the  RoboLeader  conditions. 

3.3. 1.5  SpAC 

There  were  interactions  between  SpAC  and  LOA  for  AOIs  8,  9,  and  11.  AOI  8;  high  SpAC 
participants  had  longer  duration  Glances  during  Manual  condition  than  low  SPAC.  AOI  9;  Low 
SpAC  participants  made  more  Fixations  and  Glances  during  the  RoboLeader  conditions  than 
high  SpAC  participants.  AOI  11;  Low  SpAC  participants  had  shorter  duration  Fixations  in  the 
Manual  condition  than  high  SpAC  participants.  There  were  main  effects  of  SpAC  on  fixation  and 
glance  behavior  in  AOIs  11,  12,  13,  and  14;  low  SpAC  participants  had  more  Fixations  and 
Glances  in  AOIs  12,  13,  and  14  than  high  SpAC  participants,  and  had  shorter  duration  Glances  in 
AOI  1 1  than  high  SpAC  participants. 

3.3. 1.6  SpAO 

High  SpAO  participants  had  more  Fixations  and  Glances  in  AOI  8  than  low  SpAO  participants. 
Low  SpAO  participants  had  longer  duration  glances  in  AOIs  9,  1 1,  and  14  than  those  with  high 
SpAO  scores. 

3.3. 1.7  Gaming  Experience 

Frequent  Action  Gamers  had  more  Fixations  and  Glances  in  AOIs  7  and  8  than  All  Other 
Gamers.  Frequent  Action  Gamers  had  shorter  duration  Fixations  in  AOI  11,  and  shorter  duration 
Glances  in  AOI  14,  than  All  Other  Gamers. 

3.3. 1.8  Clicking  Behavior 

Clicking  in  AOIs  10,  11,  and  12  was  predictive  of  better  performance  on  the  Target  Detection 
Task.  Frequent  Action  Gamers  clicked  in  AOI  8  more  often  than  All  Other  Gamers  in  Manual 
and  Full  Autonomous  conditions. 

Table  6  shows  overall  AOI  usage,  averaged  across  all  participants,  for  each  of  the  mission 
automation  conditions.  For  example,  Average  Dwell  Time  is  the  total  glance  time  in  that  AOI 
expressed  as  a  percentage  of  total  mission  time.  This  table  shows  not  only  how  attended  that  AOI 
was  as  compared  to  the  other  AOIs  in  any  given  mission  condition  but  also  how  changes  in 
mission  automation  condition  affected  how  a  particular  AOI  was  used.  It  is  important  to  note  that 
the  Manual  condition  had  an  additional  Vehicle  Spacing  Task  that  the  Semi-  and  Fully 
Autonomous  conditions  did  not,  so  fixations  and  glances  in  AOIs  1  and  4  may  be  artificially 
inflated  in  the  Manual  condition. 
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The  measure  “Clicked  in  AOI”  is  a  count  of  participants  who  clicked  within  an  AOI,  with  the 
premise  being  that  if  the  participant  clicked  in  the  AOI  once,  most  likely  they  continued  to  use 
that  AOI  for  the  remainder  of  that  mission.  Some  AOIs  had  no  reason  for  a  participant  to  click 
inside  that  area  (no  task  or  buttons),  and  some  AOIs  required  the  participant  to  click  in  them 
either  because  they  were  directed  to,  or  because  it  was  central  to  a  main  task.  Neither  of  those 
types  of  AOIs  were  analyzed  for  clicking  behavior.  The  AOIs  that  were  reviewed  for  clicking 
behavior  were  those  that  could  be  used  (clicked  in)  to  perform  a  task,  but  the  choice  whether  or 
not  to  utilize  that  AOI  in  that  manner  was  up  to  the  participant. 
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Table  6.  Overall  area  of  interest  (AOI)  usage,  by  LOA,  across  all  participants. 


J  Manual 

Semi-Autonomous 

Full  Autonomous  j 

Area  of  Interest 

Avg  Dwell  Time" 

Avg  Glance  Count 

Avg  Glance  Duration 
(ms) 

Avg  Fixation  Count 

Avg  Fixation 

Duration  (ms) 

Clicked  in  AOI' 

Avg  Dwell  Time" 

Avg  Glance  Count 

Avg  Glance  Duration 
(ms) 

Avg  Fixation  Count 

Avg  Fixation 

Duration  (ms) 

Clicked  in  AOI' 

Avg  Dwell  Time" 

Avg  Glance  Count 

Avg  Glance  Duration 
(ms) 

Avg  Fixation  Count 

Avg  Fixation 

Duration  (ms) 

Clicked  in  AOI' 

1  -  AOI  Map” 

37.4% 

323 

1259.6 

1282 

295.2 

NA 

19.0% 

248 

994.9 

825 

255.3 

NA 

18.3% 

233 

944.8 

765 

255.5 

NA 

2  -  AOI  Message  Center 

0.2% 

4 

328.6 

6 

193.6 

NA 

0.1% 

4 

374.1 

8 

206.6 

NA 

0.3% 

9 

379.3 

17 

183.2 

NA 

3  -  AOI  Replan  -  Cont 

0.2% 

9 

254.7 

12 

188.7 

NA 

0.2% 

9 

293.9 

12 

203.5 

NA 

0.3% 

12 

267.1 

17 

190.9 

NA 

4  -  AOI  Edit  Route  Dialogue 

1.7% 

59 

323.1 

93 

198.0 

NA 

0.3% 

11 

311.8 

18 

195.5 

NA 

0.4% 

14 

311.7 

23 

179.8 

NA 

5  -  AOI  Start  -  Undo  -  Threat 

0.4% 

14 

224.4 

18 

175.1 

NA 

0.4% 

17 

274.5 

21 

191.4 

NA 

0.4% 

16 

245.9 

22 

189.4 

NA 

6  -  AOI  Vehicle  List 

1.2% 

47 

279.1 

69 

189.5 

NA 

0.5% 

18 

261.6 

26 

196.3 

NA 

0.6% 

20 

252.9 

31 

162.4 

NA 

7  -  AOI  UAV 

1.4% 

34 

459.1 

75 

186.2 

NA 

2.4% 

54 

500.3 

127 

194.8 

NA 

2.4% 

51 

514.9 

125 

193.8 

NA 

8  -  AOI  UGV 

5.4% 

100 

551.0 

248 

213.0 

43.3% 

8.8% 

157 

651.4 

408 

230.6 

40.0% 

8.3% 

138 

672.3 

371 

231.1 

33.3% 

9  -  MGV  Front  Right 

4.2% 

113 

388.5 

197 

211.3 

76.7% 

6.3% 

160 

466.1 

317 

225.8 

80.0% 

6.3% 

158 

451.1 

309 

218.8 

86.7% 

10  -  MGV  Front  Center 

26.3% 

420 

660.8 

1010 

255.0 

90.0% 

36.0% 

550 

795.3 

1498 

270.6 

96.7% 

35.8% 

543 

753.3 

1462 

262.2 

93.3% 

1 1  -  MGV  Front  Left 

3.5% 

106 

355.4 

174 

204.9 

66.7% 

5.6% 

165 

406.4 

291 

216.6 

83.3% 

5.4% 

157 

384.1 

283 

206.7 

76.7% 

12  -  MGV  Rear  Right 

4.6% 

131 

386.5 

234 

198.8 

63.3% 

5.6% 

162 

419.2 

310 

202.9 

73.3% 

5.7% 

163 

399.8 

308 

194.9 

50.0% 

1 3  -  MGV  Rear  Center 

10.9% 

292 

395.5 

509 

216.6 

100.0% 

9.9% 

293 

415.5 

499 

226.8 

100.0% 

10.7% 

301 

431.4 

528 

217.8 

100.0% 

14  -  MGV  Rear  Left 

4.1% 

129 

345.0 

219 

195.0 

60.0% 

4.9% 

155 

381.7 

276 

200.6 

60.0% 

4.9% 

153 

367.8 

268 

194.9 

73.3% 

“Average  Dwell  Time”  is  the  percent  of  Total  Mission  time  spent  as  glances  within  that  AOI,  across  all  participants. 

Usage  data  for  Manual  condition  includes  a  Vehicle  Spacing  Task  which  the  Semi  and  Full  Autonomous  conditions  do  not. 
“Clicked  in  AOI”  is  the  percentage  of  participants  that  clicked  within  the  AOI  at  least  one  time  in  that  mission  condition. 


3.3.2  Usability  -  Findings  From  Survey,  Include  Significant  Comments 

Participants  evaluated  the  Fully  Autonomous  condition  for  perceived  usability  and  overall  trust 
in  the  system  using  the  Usability  and  Trust  Survey  (Chen  and  Barnes,  2012a).  Higher  scores 
indicated  the  Fully  Autonomous  condition  was  easier  to  use,  as  well  as  generated  more  trust  in 
the  participant  towards  the  automation.  As  such,  higher  survey  scores  were  expected  to  correlate 
with  better  performance  on  the  Target  Detection  task,  and  lower  reported  FAs.  The  Usability  and 
Trust  Survey  has  a  scoring  range  of  22-154  points.  Participant  scores  (Min  =  108,  Max  =  144, 
M=  124.75,  SD  =  9.45)  indicated  that  most  participants  found  RoboLeader  easy  to  use  and 
trusted  the  information  provided  by  the  automation. 

Usability  scores  did  not  correlate  with  performance  on  the  Target  Detection  Task  or  reported 
number  of  FAs,  nor  did  they  correlate  with  perceived  workload  (NASA-TLX  scores)  or  any 
individual  difference  factors. 

Participants  were  encouraged  to  write  any  comments  they  wished  on  the  usability  survey,  and 
those  comments  are  included  in  appendix  J. 


4.  Discussion 


The  goals  of  this  study  were  to  determine  whether  RoboLeader’s  LOA  in  managing  a  convoy 
improves  a  commander’s  ability  to  maintain  360°  security,  whether  individual  differences  (SpA, 
PAC,  and  Gaming  Experience)  have  differential  effects  on  the  commander’s  ability  to  maintain 
security,  and  whether  increasing  levels  of  assistance  resulted  in  reduced  workload  and  improved 
scanning  efficiency.  In  all  mission  conditions,  the  participants’  primary  task  was  to  identify 
threats  to  security  while  maintaining  SA.  The  level  of  automated  assistance  in  managing  the 
convoy  vehicle  spacing  and  route  varied  across  mission  conditions,  but  all  participants  were 
exposed  to  each  LOA. 

As  per  our  hypothesis,  all  participants  identified  fewer  threats  in  the  Manual  condition  and  more 
threats  in  the  RoboLeader  conditions.  Participants  with  low  SpAO  improved  their  performance 
to  a  level  similar  to  those  with  high  SpAO,  which  is  consistent  with  previous  findings  (Chen  and 
Terrence,  2009)  that  automation  can  bring  the  performance  of  persons  with  low  SpA  to  nearly 
the  same  level  as  those  with  high  SpA.  Participants  with  high  SpAO  benefited  the  most  from  a 
moderate  amount  of  automated  assistance  in  the  Semi-Autonomous  condition,  with  their 
performance  dropping  off  in  the  Fully  Autonomous  condition  to  the  same  as  those  with  low 
SpAO.  This  indicates  that,  in  the  present  scenario,  there  is  an  optimal  amount  of  automation  for  a 
multitasking  environment,  beyond  which  the  automation  does  not  improve  perfonnance  or  can 
become  detrimental  to  task  performance  for  some  users,  possibly  leading  to  automation-induced 
complacency  (Parasuraman  et  ah,  1993). 
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FAs  were  significantly  higher  in  the  Semi-Autonomous  condition  than  either  the  Manual  or  the 
Fully  Autonomous  conditions  for  all  participants.  It  is  interesting  that,  while  the  Semi- 
Autonomous  condition  seemed  to  be  the  best  balance  for  perfonnance  and  automation  for  several 
measures,  it  was  the  worst  condition  for  FAs.  While  the  participants  had  more  time  to  detect 
targets  in  the  Semi-Autonomous  condition  than  they  had  in  the  Manual  condition,  their  workload 
was  still  higher  than  in  the  Fully  Autonomous  condition,  creating  an  opportunity  for  overtrust  or 
complacency  that  resulted  in  increased  FAs  (Yeh  and  Wickens,  2000). 

Participants  with  high  SpAO  maintained  higher  SA  across  Mission  Conditions  than  those  with 
lower  SpAO.  While  RoboLeader  helped  those  with  low  SpAO  maintain  higher  SA  in  both  Semi- 
and  Fully  Autonomous  conditions,  it  did  not  appear  to  benefit  those  with  high  SpAO.  These 
results  are  consistent  with  previous  findings  (Chen  and  Terrence,  2009)  that  automation  can 
bring  the  performance  of  person’s  with  low  SpA  to  nearly  the  same  level  as  those  with  high  SpA. 

Overall,  the  results  show  that  RoboLeader  was  effective  in  improving  participants’  target 
detection  performance  and  SA  while  decreasing  perceived  workload,  regardless  of  individual 
differences.  However,  participants  with  lower  SpA  benefited  the  most  from  RoboLeader’s 
assistance,  often  bringing  their  perfonnance  near  the  same  as  those  with  higher  SpA. 

LOA  affected  perceived  workload,  with  each  successive  increase  in  autonomy  showing  a 
decrease  in  perceived  workload  as  reported  using  the  NASA-TLX.  However,  physiological 
eyetracking  data  did  not  fully  support  the  finding  of  the  NASA-TLX;  Blink  Rate  and  Fixation 
Count  indicated  the  RoboLeader  conditions  were  nearly  equivalent  on  workload,  while  the 
Manual  condition  induced  the  highest  level  of  workload.  Nonetheless,  as  workload  decreased,  so 
did  the  average  length  of  saccades,  indicating  more  efficient  scanning  patterns  in  the  Semi- 
Autonomous  condition  than  in  the  Manual  condition.  Previous  studies  have  differed  on  perceived 
workload  as  well;  RoboLeader’s  presence  did  not  appear  to  affect  perceived  workload  in  Chen 
and  Bames  (2012a),  however,  in  Chen  and  Barnes  (2012b)  participants  reported  lower  workload 
when  assisted  by  RoboLeader.  It  is  possible  that  the  difference  in  reported  workload  between  the 
RoboLeader  conditions  is  actually  a  difference  in  participant  engagement  (i.e.,  boredom). 

Average  Pupil  Diameter  was  largest  in  the  Manual  condition — indicating  it  was  most  taxing  and 
decreased  for  each  successive  increase  in  automation,  which  is  evidence  that  each  additional 
LOA  made  the  trial  less  difficult  or  interesting  (Van  Orden  et  al.,  2000,  2001;  Iqbal  et  al.,  2004; 
Kang  et  al.,  2009),  and  this  finding  agrees  with  the  reported  NASA-TLX  scores.  However,  pupil 
dilation  depended  on  PAC  individual  differences.  Participants’  measured  Average  Pupil 
Diameter  was  divided  along  Attentional  Control  scores;  those  with  low  PAC  had  consistent  Pupil 
Diameter  regardless  of  LOA,  while  Pupil  Diameter  of  the  high  PAC  participants  decreased  with 
each  successive  LOA,  as  well  as  being  considerably  larger  than  the  low  PAC  groups’  Pupil 
Diameters  in  all  conditions.  Low  PAC  participants  had  higher  Fixation  Counts  across  LOA  than 
high  PAC  participants,  which  when  considered  with  the  Pupil  Diameter  findings  would  suggest 
that  low  PAC  participants  were  working  at  their  highest  capacity,  and  as  such,  could  not  keep  up 
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with  the  task  demands,  resulting  in  a  leveling  off  of  Pupil  Diameter  changes  (Peavler,  1974). 
However,  Blink  Rate,  Fixation  Count,  and  Saccadic  Amplitude  were  not  significantly  different 
between  the  two  RoboLeader  conditions,  whereas  the  TLX  scores  suggest  at  least  a  perceived 
difference  among  the  conditions.  The  individual  differences  in  PAC  suggest  that  high  PAC 
participants  spent  the  least  effort  on  the  highest  automation  level  without  a  concomitant  increase 
in  perfonnance  compared  to  the  low  PAC  participants.  More  telling,  as  mentioned  above, 
individuals  with  the  highest  scores  on  SpAO  showed  a  decreased  hit  rate  for  targeting  for  the 
highest  LOA  compared  to  the  mid-level,  implying  too  much  automaton  may  be  detrimental  for 
highly  skilled  operators  (see  Parasuraman  et  ah,  2009). 

4.1  AOI  Usage 

A  secondary  goal  of  this  study  was  to  evaluate  the  utility  of  the  OCU  in  general,  and  various 
camera  feeds  specifically,  in  perfonning  the  Target  Detection  Task  and  maintaining  SA. 
Differential  effects  of  Spatial  Ability,  PAC,  and  Gaming  Experience  on  utility  were  also 
examined. 

The  tendency  for  an  operator  to  center  their  vision  on  a  central  area  of  a  display  is  well 
documented,  and  the  farther  from  the  center  an  object  or  area  is,  the  less  likely  operators  will 
attend  to  it  (Tatler,  2007).  However,  in  order  for  the  participant  to  perfonn  well  in  this 
experiment,  they  were  required  to  visually  monitor  the  periphery  of  the  display.  How  this  is 
being  done  and  who  is  doing  it  is  of  great  interest  to  display  designers,  as  well  as  those  who 
would  train  potential  operators. 

AOI  2  was  the  incoming  message  center,  where  the  participant  would  receive  messages  either 
identifying  a  threat  to  avoid  or  investigate.  When  workload  was  high,  participants  did  not  rely  on 
AOI  2  for  this  infonnation,  instead  they  relied  on  using  the  color-coded  information  that 
appeared  on  their  interactive  map  to  determine  what  response  was  required  of  them.  This  appears 
to  be  an  example  of  successful  dual  notification  systems;  when  resources  were  limited,  the 
participants  were  still  able  to  receive  and  correctly  interpret  the  meaning  of  the  message.  This 
experiment  had  only  two  potential  incoming  messages,  more  complex  or  diverse  messages  may 
result  in  interpretation  failure. 

AOI  7  was  the  UAS  camera  feed  in  the  upper  left  comer  of  the  OCU  display,  and  it  was  expected 
that  participants  who  maintained  high  SA  would  monitor  this  camera  feed  for  information. 
However,  overall  glance  behavior  in  this  AOI  did  not  predict  SA.  Frequent  Action  Gamers  and 
participants  with  high  SpAO  scored  higher  on  SA  measures  than  other  participants,  but  of  the 
two,  only  the  Frequent  Action  Gamers  actively  monitored  this  AOI  across  all  automation  levels. 
This  implies  that  while  this  information  could  be  useful  for  maintaining  SA,  practice  and 
experience  are  needed  to  teach  operators  how  to  use  this  information. 
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AOI  8  was  the  UGV  camera  feed,  located  upper  mid-screen  on  the  left  side  of  the  display.  Views 
in  this  camera  feed  gave  the  participant  an  advanced  view  of  their  route;  approximately  50 
meters  ahead  of  the  manned  vehicle,  and  the  participant  could  mark  targets  using  this  camera 
feed.  It  was  expected  that  participants  who  scored  well  on  the  Target  Detection  Task,  and  those 
that  had  the  fewest  FAs,  would  utilize  this  AOI.  Participants  who  had  the  longest  Average 
Fixations  in  this  AOI  did  perfonn  better  on  the  Target  Detection  Task,  but  only  in  the  Fully 
Autonomous  condition.  There  were  fewer  Fixations  and  Glances  in  AOI  8  in  the  Manual 
condition  than  in  the  RoboLeader  conditions,  across  all  participants,  indicating  that  when 
workload  increased  this  AOI  was  not  utilized  as  often.  Participants  who  were  not  Frequent 
Action  Gamers  were  less  likely  to  use  this  AOI  to  mark  targets,  regardless  of  automation  level. 

AOIs  9,  10,  and  1 1  comprised  the  MGV  forward  camera  feed,  and  it  was  expected  that  this  feed 
would  be  used  for  maintaining  SA  and  conducting  the  Target  Detection  Task.  Overall  Fixation 
and  Glance  behavior  in  AOIs  9,  10,  and  1 1  were  not  predictive  of  perfonnance  in  the  Target 
Detection  Task  or  of  reported  FAs.  Frequent  Action  Gamers  had  significantly  shorter  duration 
Fixations  and  Glances  in  AOIs  10,  and  11,  while  low  SpAC  and  low  SpAO  participants  had 
longer  and  more  frequent  glances  in  AOIs  9,  10,  and  11.  This  implies  that  those  participants  that 
have  developed  more  advanced  scanning  strategies  relied  on  information  from  these  AOIs  less 
than  other  participants  did,  while  participants  with  poor  SpA  relied  on  the  forward  camera  feeds 
more  than  participants  with  high  SpA. 

AOIs  12,  13,  and  14  comprised  the  MGV  rearward  camera  feed,  and  it  was  expected  that  this 
feed  would  be  used  for  maintaining  SA  and  conduction  the  Target  Detection  Task.  Overall 
Fixation  and  Glance  behavior  in  AOIs  12,  13,  and  14  were  not  predictive  of  performance  in  the 
Target  Detection  Task.  Overall  Fixation  and  Glance  behavior  in  AOI  12  were  predictive  of 
reported  FAs  in  the  Manual  condition.  Frequent  Action  Gamers  and  high  SpAO  participants  had 
significantly  shorter  duration  Fixations  and  Glances  in  AOI  14,  while  low  SpAC  participants  had 
more  frequent  Fixations  and  Glances  in  AOIs  12,  13,  and  14.  Participants  with  more  developed 
scanning  strategies  optimized  their  glance  behavior  in  these  AOIs,  while  participants  with  poorer 
SpA  attempted  to  monitor  this  feed  (as  indicated  by  the  high  number  of  fixations),  however,  they 
did  not  necessarily  gain  better  information  from  their  attempts  (as  indicated  by  the  higher 
number  of  FAs).  Most  participants  did  incorporate  AOIs  12,  13,  and  14  into  their  Target 
Detection  strategy,  however,  usage  decreased  as  the  LOA  decreased. 


5.  Conclusions 


In  the  current  study,  we  investigated  whether  increasing  RoboLeader’ s  LOA  improved  an 
operator’s  ability  to  maintain  360°  security  and  SA,  reduced  workload  levels  and  improved  scan 
efficiency,  as  well  as  the  effect  of  individual  differences  on  performance  across  LOAs.  Overall, 
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increasing  LOA  did  improve  perfonnance  in  the  Target  Detection  Task  and  result  in  lower 
reported  workload;  however,  there  appear  to  be  differential  effects  due  to  individual  differences 
that  would  suggest  an  optimal  level  of  assistance  exists.  We  see  a  perfonnance  drop  on  the  target 
detection  task  for  the  high  SpAO  group  in  the  Full  Autonomous  condition.  Although  the  low 
SpAO  group  improves  slightly  with  each  increasing  LOA,  the  loss  in  perfonnance  of  the  high 
SpAO  group  is  greater  than  the  gain  of  the  low  SpAO  in  the  Full  Autonomous  condition.  Thus, 
overall  performance  on  the  target  detection  task  depended  in  individual  differences  in  spatial 
ability  as  well  as  LOA.  Some  of  the  eye-tracking  behavior  data  suggest  this  could  be  due  to 
skilled  operators’  disengagement  at  the  highest  LOA. 

Increasing  LOA  did  reduce  reported  workload,  with  each  increasing  LOA  significantly  reducing 
workload.  However,  eye  behavior  data  indicates  there  were  differential  effects  due  to  PAC,  with 
participants  with  high  PAC  able  to  benefit  from  the  increasing  LOA  while  those  with  low  PAC 
struggling  regardless  of  LOA. 

Usability  analysis  of  the  OCU  demonstrated  that  the  participants  with  high  SpA  and  Frequent 
Action  Gamers  were  better  able  to  utilize  the  entire  display  than  their  counterparts,  and  results 
indicate  this  led  to  improved  performance  and  SA  scores.  Increased  LOA  also  resulted  in  better 
display  utilization. 

Future  research  should  investigate  how  dependent  this  seeming  “optimal”  level  of  assistance  is 
on  task  number  and  type,  as  well  as  explore  OCU  designs  which  could  be  equally  effective 
regardless  of  the  operators’  scanning  expertise  and/or  skill.  Personnel  testing  and  training  should 
also  be  explored,  so  that  operators  with  low  SpA  and  PAC  can  perfonn  at  similar  levels  to  those 
with  high  SpA  and  PAC.  In  addition,  automation  itself  seems  to  be  an  equalizer  for  low  spatial 
abilities  as  well  as  low  PAC,  with  the  caveat  that  high  levels  can  be  detrimental  to  more  highly 
skilled  individuals  because  the  automated  tasks  fail  to  capture  their  full  attention.  The  literature 
on  adaptive  automation  argues  that  automation  can  be  modulated  to  optimally  engage  the 
operator  depending  on  task  difficulty;  these  data  suggest  that  adaptation  should  be  dependent  on 
the  operator’s  skill  level  as  well  (Chen  et  ah,  2011;  Parasuraman  et  ah,  2009) 
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Appendix  A.  Demographics  Questionnaire 


This  appendix  appears  in  its  original  form,  without  editorial  change. 
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Demographic  Questionnaire 


Participant* _  Age _  Major _  Date _  Gender _ 

1 .  What  is  the  highest  level  of  education  you  have  had? 

Less  than  4  yrs  of  college _  Completed  4  yrs  of  college _ Other _ 

2.  When  did  you  use  computers  in  your  education?  (Circle  all  that  apply) 

Grade  School  Jr.  High  High  School 

Technical  School  College  Did  Not  Use 

3.  Where  do  you  currently  use  a  computer-1  (Circle  all  that  avviv) 

Home  Work  Library  Other _  Do  Not  Use 

4.  For  each  of  the  following  questions,  circle  the  response  that  best  describes  you. 

How  often  do  you: 

Use  a  mouse?  Daily.  Weekly,  Monthly,  Once  every  few  months,  Rarely,  Never 

Use  a  joystick?  Daily,  Weekly,  Monthly,  Once  every  few  months.  Rarely,  Never 

Use  a  touch  screen?  Daily,  Weekly,  Monthly,  Once  every  few  months,  Rarely,  Never 

Use  icon-based  programs  software1 

Daily,  Weekly,  Monthly,  Once  every  few  months,  Rarely,  Never 
Use  programs  software  with  pull-down  menus? 

Daily,  Weekly,  Monthly,  Once  every  few  months,  Rarely,  Never 
Use  graphics  drawing  features  in  software  packages? 

Daily,  Weekly,  Monthly,  Once  every  few  months,  Rarely,  Never 
Use  E-mail?  Daily,  Weekly,  Monthly,  Once  every  few  months,  Rarely,  Never 

Operate  aradio  controlled  vehicle  fear,  boat,  or  plane)? 

Daily,  Weekly,  Monthly,  Once  every  few  months,  Rarely,  Never 

Play  computer  video  games1 

Daily,  Weekly,  Monthly,  Once  every  few  months,  Rarely,  Never 

5.  Which  type(s)  of  computer  video  games  do  youmost  often  play  if  you  play  at  least  once  every  few  months? 

6.  Which  of  the  following  best  describes  your  expertise  with  computer?  (check  V  one) 

_ Novice 

_ Good  with  one  type  of  software  package  (such  as  word  processing  or  slides) 

_ Good  with  several  software  packages 

_ Can  program  in  one  language  and  use  several  software  packages 

_ Can  program  in  several  languages  and  use  several  software  packages 

7.  Are  you  in  your  good,  comfortable  state  of  health  physically?  YES  NO 
If  NO,  please  briefly  explain: 

8.  How  many  hours  of  sleep  did  you  get  last  night? _ hours 

9.  Do  you  have  normal  color  vision?  YES  NO 

10.  Do  you  have  prior  military  service?  YES  NO  If  Yes,  how-  long _ 
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Appendix  B.  Attentional  Control  Survey 


This  appendix  appears  in  its  original  form,  without  editorial  change. 


41 


Attentional  Control  Survey  Participant# _  Date _ 

For  each  of  the  following  question:,  circle  the  response  that  best  describes  you. 

It  is  very  hard  for  me  to  c  oncer. tote  on  a  difficult  task  when  there  are  noises  around. 

Almost  never,  Sometimes,  Often,  Always 


When  I  need  to  concaitote  and  solve  a  problem,  I  have  trouble  focusing  my  attention. 

Almost  never,  Sometimes,  Often,  Always 


When  I  am  working  hard  on  something,  I  still  get  distracted  by  events  around  me. 

Almost  ne%-er,  Sometimes,  Often,  Always 


My  concentration  is  good  even  if  there  is  music  in  the  room  around  me. 

Almost  never,  Sometimes,  Often,  Always 


When  concentrating,  I  can  focus  my  attention  so  that  I  become  unaware  of  what's  going  on  in  theroom  around  me. 

Almost  never,  Sometimes,  Often,  Always 


When  I  am  reading  or  studying,  I  am  easily  distracted  if  there  are  people  talking  in  the  same  room. 

Almost  never,  Sometimes,  Often,  Always 


When  trying  to  focus  my  attention  on  something,  I  have  difficulty  blocking  out  distracting  thoughts. 

Almost  never,  Sometimes,  Often,  Always 

I  have  a  hard  time  concentrating  when  I'm  excited  about  something. 

Almost  never,  Sometimes,  Often,  Always 

When  concentrating,  I  ignore  feelings  of  hunger  or  thirst.  Almost  never,  Sometimes,  Often,  Always 

I  can  quickly  switch  from  one  task  to  another.  Almost  never,  Sometimes.  Often,  Always 

It  takes  me  a  w-hile  to  get  really  involved  in  a  new  task.  Almost  never,  Sometimes,  Often,  Always 

It  is  difficult  for  me  to  coordinatemy  attention  between  the  listaiing  and  writing  required  when  takingnotes  during 
lectures.  Almost  never,  Sometimes,  Often,  Always 

I  can  become  interested  in  a  new  topic  very  quickly  when  I  need  to. 

Almost  never,  Sometimes,  Often,  Always 

It  is  easy  for  me  to  read  or  write  while  I'm  also  talking  on  the  phone. 

Almost  never,  Sometimes,  Often,  Always 

I  have  trouble  carrying  on  two  conversations  at  once.  Almost  never,  Sometimes,  Often,  Always 

I  have  a  hard  time  coming  up  with  new  ideas  quickly.  Almost  never.  Sometimes,  Often,  Always 

After  being  interrupted  or  distracted,  I  can  easily  shift  my  attention  back  to  what  I  was  doing  before. 

Almost  never,  Sometimes,  Often,  Always 

When  a  distracting  thought  comes  to  mind,  it  is  easy  for  me  to  shift  my  attention  away  from  it. 

Almost  never,  Sometimes,  Often,  Always 

It  is  easy  for  me  to  alternate  between  two  different  tasks.  Almost  never,  Sometimes,  Often,  Always 


It  is  hard  for  me  to  break  from  one  way  of  thinking  about  something  and  look  at  it  from  another  point  of  view. 

Almost  never,  Sometimes,  Often,  Always 
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Appendix  C.  Cube  Comparison  Test 


This  appendix  appears  in  its  original  form,  without  editorial  change. 


Cube  Comparisons  Test 


Participant  # 


Date 


CUBE  COMPARISONS  TEST  —  S-2  (Rev.) 

Wooden  blocks  such  as  children  play  with  are  often  cubical  with  a  different 
letter,  number,  or  symbol  on  each  of  the  six  faces  (top,  bottom,  four  sides). 

Each  problem  in  this  test  consists  of  drawings  of  pairs  of  cubes  or  blocks  of 
this  kind.  Remember,  there  is  a  different  design,  number,  or  letter  on  each  face 
of  a  given  cube  or  block.  Compare  the  two  cubes  in  each  pair  below. 


S  t — y  D  wmm  S  mm*  D  i — i 


The  first  pair  is  marked  D  because  they  must  be  drawings  of  different  cubes. 

If  the  left  cube  is  turned  so  that  the  A  is  upright  and  facing  you,  the  N  would  be 
to  the  left  of  the  A  and  hidden,  not  to  the  right  of  the  A  as  is  shown  on  the  right 
hand  member  of  the  pair.  Thus,  the  drawings  must  be  of  different  cubes. 

The  second  pair  is  marked  S  because  they  could  be  drawings  of  the  same  cube. 
That  is,  if  the  A  is  turned  on  its  side  the  X  becomes  hidden,  the  B  is  now  on  top, 
and  the  C  (which  was  hidden)  now  appears.  Thus  the  two  drawings  could  be  of  the 
same  cube. 

Note:  No  letters,  numbers,  or  symbols  appear  on  more  than  one  face  of  a  given 

cube.  Except  for  that,  any  letter,  number  or  symbol  can  be  on  the  hidden  faces  of 
a  cube. 


Work  the  three  examples  below. 


S  C=J  DCZD  S  f=)  DO  S  C=3  DCZl 


The  first  pair  immediately  above  should  be  marked  D  because  the  X  cannot  be  at 
the  peak  of  the  A  on  the  left  hand  drawing  and  at  the  base  of  the  A  on  the  right 
hand  drawing.  The  second  pair  is  "different"  because  P  has  its  side  next  to  G  on 
the  left  hand  cube  but  its  top  next  to  G  on  the  right  hand  cube.  The  blocks  in  the 
third  pair  are  the  same,  the  J  and  K  are  just  turned  on  their  side,  moving  the  O  to 
the  top. 

Your  score  on  this  test  will  be  the  number  marked  correctly  minus  the  number 
marked  incorrectly.  Therefore,  it  will  not  be  to  your  advantage  to  guess  unless  you 
have  some  idea  which  choice  is  correct.  Work  as  quickly  as  you  can  without  sacri¬ 
ficing  accuracy. 

You  will  have  3  minutes  for  each  of  the  two  parts  of  this  test.  Each  part  has 
one  page.  When  you  have  finished  Part  1,  STOP. 

DO  NOT  TURN  THE  PAGE  UNTIL  YOU  ARE  ASKED  TO  DO  SO. 

Copyright  (c^  1962,  1976  by  Educational  Testing  Service.  All  rights  reserved. 
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Appendix  D.  Spatial  Orientation  Test 


This  appendix  appears  in  its  original  form,  without  editorial  change. 
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The  Spatial  Orientation  Test,  modeled  after  the  cardinal  direction  test  developed  by  Gugerty  and 
his  colleagues  (Gugerty  &  Brooks,  2004),  is  a  computerized  test  consisting  of  a  brief  training 
segment  and  32  test  questions.  The  program  automatically  captures  both  accuracy  and  response 
time.  Participants  are  shown  the  following  image: 


The  right  side  image  is  of  a  map  showing  a  plane  flying.  The  left  side  of  the  display  is  the  pilot’s 
view  (from  the  cockpit  of  the  plane)  of  several  parking  lots  surrounding  a  building.  The 
participants’  task  is  to  use  the  right  side  of  the  display  to  learn  in  which  direction  the  plane  is 
flying.  They  then  use  this  information  to  identify  which  parking  lot  (north,  south,  east,  or  west) 
in  the  left  side  image  has  the  dot.  In  the  example  shown  above,  the  plane  is  heading  north,  and  so 
the  dot  appears  in  the  north  parking  lot.  In  the  example  shown  below,  the  plane  is  heading  south, 
and  so  the  dot  appears  in  the  east  parking  lot. 


Participants  are  shown  32  of  these  images  in  succession;  each  time  the  direction  the  plane  is 
flying  and  the  location  of  the  dot  are  randomized.  Participants  answer  by  clicking  on  one  of  four 
buttons  (North,  South,  East,  or  West).  This  test  is  self-paced;  the  participant  may  take  as  long  as 
they  wish  to  answer,  and  when  they  answer  one  question  the  next  question  automatically 
appears.  No  questions  can  be  skipped,  and  the  order  of  images  is  randomized  among  participants. 
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Appendix  E.  NASA-TLX  Questionnaire 


This  appendix  appears  in  its  original  form,  without  editorial  change. 


NASA  TLX  Questionnaire 


NASA  TLX  Workload  .Assessment 


Instructions:  Eatings  Scales 

We  are  interested  in  the  'workload"  yon  experienced  during  this  scenario.  Workload  is  something 
experienced  individually  by  each  person.  One  way  to  find  out  about  workload  is  to  ask  people  to  describe 
what  they  experienced.  Workload  mas-  be  caused  by  many  different  factors  and  tie  would  like  you  to 
evaluate  them  individually-  The  set  of  six  workload  ratine  factors  was  developed  for  you  to  use  in 
evaluating  your  experiences  during  different  tasks.  Please  read  them.  If  you  have  a  question  shout  any  of 
the  scales  in  the  table,  please  ask  about  it  It  is  extremely  important  that  they  be  dear  to  you. 


Definitions 


Title 

Endpoints 

Descriptions 

MENTAL  DEMAND 

Low  Him 

How  much  menial  and  perceptual  activity  was  required  (that 

is.  thinking,  deciding,  calculating  remembering  looking 
searching  etc  )?  Was  the  task  easy  or  demanding,  simule  or 
complex,  exacting  or  forgiving" 

PHYSICAL  DEMAS"D 

Low  High 

How  much  physical  activity  was  required  (to  is.  pushing, 
pulling  taming  controlling  activating  etc  )?  Was  the  task 
easv  or  demanding  slow  or  brisk,  slack  or  strenuous,  restful 
or  laborious? 

TEMPORAL  DEMAND 

Low  High 

How  much  time  pressure  did  you  feel  due  to  the  rate  or  pace 
ax  which  the  tasks  or  task  elements  occurred?  Was  the  pace 
slow  and  leisurely  or  rapid  and  fiaatk? 

PERFORMANCE 

Poor  C-ood 

How  successful  do  you  thick  you  were  in  accomplishing  the 
goals  of  the  task"  How  satisfied  were  you  with  your 
performance  in  accomplishingthese  goals? 

EFFORT 

Low  Hi  21 

How  hard  did  you  have  to  work  (menially  and  phwkally)  to 
accomplish  your  level  of  performance? 

FRUSTRATION  LEVEL 

Low  Him 

How  insecure,  discouraged,  irritated,  stressed,  and  annoyed 
versus  secure,  zatified.  content,  relaxed  and  complacent  did 
you  feel  during  the  task7 

We  want  you  to  evaluate  workload.  Rate  the  workload  on  each  factor  on  a  scale.  Each  scale  has 
two  and  descriptions,  and  20  slots  (hash marks)  between  the  end  descriptions.  Place  an  t:x':  in  the 
slot  (between  the  hash  marks)  that  you  feel  most  accurately  reflects  your  workload. 

After  you  have  finished  the  entire  series,  we  will  be  able  to  use  the  pattern  of  your  choices  to  create  a 
weighted  combination  of  ratings  into  a  summary  workload  score. 

We  ask  you  to  evaluate  your  workload  for  this  scenario.  This  indudes  all  the  duties  involved  in  your  job 
(e.g.,  detecting  targets  and  'using  display). 
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Participant  ID: 


TLX  Workload  Scale 

Please  rate  your  workload  by  putting  a  mark  on  each  of  the  six  scales  at  the  point  which  matches  your 
experience. 


Men  ral  Demand 


I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I 


Low 


Hglh 


Physical  Demand  |  |  |  |  |  |  |  |  |  | 

Low  High 


Temporal  Demand 


I  I  I  I  I  I  I  I  I 


I  I  I  I  I  I  I  I  I 


Low 


High 


Peifoimanee 


I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I  I 


Good 


Poor 


EffoiT 
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Appendix  F.  Usability  and  Trust  Survey 


This  appendix  appears  in  its  original  form,  without  editorial  change 
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Participant 


Date. 


Usability  &  Trust  Survey  Comments 

L  I  found  die  camera  feeds  of  UAS,  JJVG  and  MGY  to  be  helpful  during  route  modification. 

Strongly  DISAGREE  | - i — j  — 1  Strongly  AGREE  N'A 

1  2  3  4  5  6  7 

2.  I  made  use  of  the  RoboLeader's  recommendations. 

Stronslv  DISAGREE  | — j — | — | — | — 1 — |  StronElv  AGREE  N/A 

1  2  3  4  5  6  7 

3.  I  sometimes  felt ‘lost1  using  the  RoboLeader  display. 

Strongly  DISAGREE  | — i - | — | — I — 1  Strongly  AGREE  N'A 

1  2  3  4  5  6  7 

4.  The  RoboLeader  display  was  intuitive  and  made  it  easy  to  determine  how  to  edit  routes. 

Stronslv  DISAGREE  |^— | — | — | — | — | — |  StronElv  AGREE  N/A 

1  2  3  4  5  6  7 

5.  I  do  not  feel  the  RoboLeader  display  was  helpful  in  the  task. 

StronElv  DISAGREE  | — | - 1 — | — | — |  StronElv  AGREE  N/A 

1  2  3  4  5  6  7 

6.  I  relied  heavily  on  the  RoboLeader  for  the  task. 

StronElv  DISAGREE  | — | — | — I — | — i — j  Stronslv  AGREE  N/A 

1  2  3  4  5  6  7 

7.  Threats  were  visible  on  die  screen(s)  long  enough  to  accurately  detect  them. 

StronElv  DISAGREE  | — | — | — | — 1 — | — |  StronElv  AGREE  N'A 

1  2  3  4  5  6  7 

8.  The  RoboLeader  display  w  as  confusing. 

Strongly  DISAGREE  [ — i — | — j — j — j — j  Strongly  AGREE  N'A 

1  2  3  4  5  6  7 

9.  The  RoboLeader  display  w  as  annoying. 

StronElv  DISAGREE  | — | - 1 — 4 — | — |  StronElv  AGREE  N'A 

1  2  3  4  5  6  7 

10.  The  RoboLeader  display  improved  nry  performance  on  the  task. 

Strongly  DISAGREE  [ — | - ) — | — I —  Strongly  AGREE  N'A 

1  2  3  4  5  6  7 

11.  The  RoboLeader  display  can  be  deceptive. 

Stronslv  DISAGREE  | — | — | — | — | — | — |  StronElv  AGREE  N'A 

1  2  3  4  5  6  7 

12.  The  RoboLeader  display  sometimes  behaves  in  an  unpredictable  manner. 

Strongly  DISAGREE  | — | — i — j — H — |  — i  Strongly  AGREE  N'A 

1  2  3  4  5  6  7 
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Participant 


Date. 


Usability  &  Trust  Survey  Comments 

13.  I  am  often  suspicious  of  theRoboLeader  system's  intent,  action  or  outputs. 

Strondv  DISAGREE  | — | — | — | — | — | — |  Stronglv  AGREE  K'A 

1  2  3  4  5  6  7 

14.  I  am  sometimes  unsure  of  theRoboLeader  system. 

Strondv  DISAGREE  | - - — i — i - Strondv  AGREE  K'A 

1  2  3  4  5  6  7 

1$.  The  RoboLeader  system  may  have  harmful  effects  on  the  task. 

Strondv  DISAGREE  | — — j — | — ! — i — |  Strondv  AGREE  NA 

1  2  3  4  5  6  7 

16.  I  am  confident  in  the  RoboLeader  ss'stem. 

S trend v  DISAGREE  | - 1 — | — j  —  Strondv  AGREE  K'A 

1  2  3  4  5  6  7 

17.  The  RoboLeader  system  can  proside  security. 

S trend v  DISAGREE  | — — | — | 1 — | — 1  Stronglv  AGREE  K'A 

1  2  3  4  5  6  7 

18.  The  RoboLeader  ss'stem  has  integrity. 

S trend v  DISAGREE  | - 1 -  Strondv  AGREE  K'A 

1  2  3  4  5  6  7 

19.  The  RoboLeader  ss'stem  is  dependable. 

Strongly  DISAGREE  | — | — | — | — | — | — |  Strongly  AGREE  K'A 

1  2  3  4  5  6  7 

20.  The  RoboLeader  system  is  consistent 

Strondv  DISAGREE  | — - 1 — ; — j  —  Strondv  AGREE  K'A 

1  2  3  4  5  6  7 

21.  I  can  trust  die  RoboLeader  ss'stem. 

Strongly  DISAGREE  | — — — \ — | - Strongly  AGREE  K  A 

1  2  3  4  5  6  7 

22.  I  am  familiarsvith  the  RoboLeader  display. 

Stronglv  DISAGREE  | — | — | — | — | — | — |  Strondv  AGREE  K'A 

1  2  3  4  5  6  7 
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Appendix  G.  Situation  Awareness  Questions 


This  appendix  appears  in  its  original  form,  without  editorial  change. 
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Example  3  SA  Questions 

1)  Use  the  provided  paper  to  identify  a  route  that  was  edited  to  avoid  a  fire 

2)  What  was  the  compass  direction  (North,  South,  East,  West)  of  your  MGV  prior  to  your  last 

turn? 

Mission  1  SA  Questions 

1)  Use  the  provided  paper  to  highlight  an  area  that  currently  has  a  fire 

2)  Use  the  provided  paper  to  identify  where  your  UGV  is  currently  located 

3)  Use  the  provided  paper  to  identify  one  route  that  was  edited  to  perform  reconnaissance  in  an 

Area  of  Interest 

4)  Use  the  provided  paper  to  identify  the  most  recent  change  made  to  the  route 

5)  What  is  currently  happening  in  area  X  of  your  map? 

Mission  2  SA  Questions 

1)  What  was  your  MGV's  compass  direction  (North,  South,  East,  West)  prior  to  this  blank 

screen? 

2)  Use  the  provided  paper  to  identify  the  most  recent  change  made  to  the  route 

3)  Use  the  provided  paper  to  identify  one  route  that  was  edited  to  avoid  a  Hostile  Area 

4)  Use  the  provided  paper  to  highlight  one  Area  of  Interest 

5)  Use  the  provided  paper  to  identify  one  route  that  has  encountered  smoke 

Mission  3  SA  Questions 

1)  Use  the  provided  paper  to  identify  where  your  UGV  is  currently  located 

2)  What  was  the  compass  direction  (North,  South,  East,  West)  of  your  MGV  prior  to  your  last 

turn? 

3)  Use  the  provided  paper  to  highlight  one  Hostile  Area 

4)  Use  the  provided  paper  to  identify  a  route  that  was  edited  to  avoid  a  fire 

5)  What  is  currently  happening  in  area  X  of  your  map? 
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Appendix  H.  Participant  SA  Query  Answer  Sheets 


This  appendix  appears  in  its  original  form,  without  editorial  change. 
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Participant: 


Date: 


Practice  BJi  M 


I 


Vehicle  Spacing: 

UAS  (Raven)  to  UGV:  200  m 
UGVtoMGV:50  m 
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Participant: 


Date: 


Mission  1  BJi  M 


Maximum  Vehicle  Spacing: 
UAS  (Raven)  to UGV:  200m 
UGVtoMGV:50m 


1. 


2. 


3. 


4. 


5. 
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Partidparrt: 


Date: 


Mission  2  8_.S,  M 


Jvlaxinnum  vehicle  Spadrsg: 
UAS  (Raven)  to  UGV:  200  m 
UGV  to  MGV:  50  m 


1_ 


2. 


3. 


4. 


5. 
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Participant: 


Date: 


MaxTijn  V etude  Spacing: 
UAS  (Raven)  to  DGV:  200  m 
UGV  to  MGV:  50  m 


Mission  3  R,S,  M 


N 


s 
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Intentionally  left  blank. 
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1.1  AOI  Analysis  Method 

In  order  to  analyze  participants’  Area  of  Interest  (AOI)  usage,  the  following  measures  and 
behaviors  were  collected  by  eyetracker: 

•  Overall  mission  measures: 
o  Total  Time  of  mission 
o  Total  Dwell  Time 

•  AOI  specific  measures  (by  mission): 
o  Total  Dwell  Time 

o  Number  of  Glances  (Dwells) 
o  Number  of  Fixations 
o  Total  Fixation  Time 
o  Duration  of  First  Fixation 

o  Entry  Time  of  First  Fixation  (as  mission  time  elapsed) 
o  Sequence  of  First  Fixation  among  all  AOIs 
o  Time  to  First  Mouse  Click 

Several  additional  measures  were  calculated  from  the  observed  measures: 

•  Average  Glance  Duration  (per  AOI,  by  mission)  -  AOI  Total  Dwell  Time  divided  by  AOI 
Number  of  Glances,  calculated  for  each  mission  condition. 

•  Average  Fixation  Duration  (per  AOI,  by  mission)  -  AOI  Total  Fixation  Time  divided  by 
AOI  Number  of  Fixations,  calculated  for  each  mission  condition. 

•  Yes/No:  did  participant  mouse  click  in  this  AOI? 

Each  AOI  section  has  a  brief  explanation  of  its  purpose  and  contents,  expected  participant 
interaction,  and  which  analyses,  if  any,  would  be  appropriate  or  infonnative  for  that  particular 
AOI.  Not  all  factors  are  examined  for  all  AOIs.  With  some  minor  variation,  analysis  of  each  AOI 
was  conducted  in  the  following  manner: 

1 .  Repeated  measures  (within-subject)  ANOVA  was  used  to  evaluate  differences  in  Fixation 
and  Glance  behavior  within  the  AOI  due  to  mission  condition. 

2.  Simple  Linear  Regression  was  used  to  evaluate  if  Fixation  and  Glance  behavior  was 
predictive  of  Target  Detection  Task  performance,  reported  FAs,  Perceived  Workload 
and/or  SA. 
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3.  Individual  difference  factors  Gaming  Experience,  PAC,  SpAO,  and  SPAC  were  evaluated 
for  correlations  with  Fixation  and  Glance  behavior. 

a.  A  mixed  between-within  subjects  ANOVA  was  used  to  further  investigate  and  define 
significant  correlations  found  in  step  3. 

4.  Clicking  Behavior  was  evaluated  for  correlation  with  Target  Detection,  FAs  and  Individual 
Difference  factors.  Clicking  behavior  for  all  participants  is  reported  as  percentages. 

a.  Significant  correlations  for  2  x  2  tables  (Individual  Difference  factors)  were  verified 
using  Chi-squared  analysis,  Yates’  Correction  for  Continuity. 

b.  Significant  correlations  for  continuous  data  (Target  Detection  and  FAs)  were  further 
defined  using  Simple  Finear  Regression. 

1.2  AOI 1  Map 

AOI  1  (Map)  was  expected  to  have  a  high  number  of  fixations  and  dwells,  due  to  its  size  (21.5% 
of  total  screen  area),  position,  and  utility.  The  participant  could  manipulate  the  view  displayed  in 
this  AOI,  and  this  AOI  was  used  to  manage  the  convoy  route  and  vehicle  spacing,  as  well  as  to 
gather  infonnation  needed  to  maintain  SA.  The  number  of  fixations  and  total  dwell  time  in  this 
AOI  was  expected  to  vary  with  mission  condition,  with  the  highest  number  being  in  the  Manual 
condition  due  to  an  additional  task  (Vehicle  Spacing)  that  was  automated  in  the  RoboFeader 
conditions  and,  as  such,  was  perfectly  confounded  with  condition.  For  these  reasons,  AOI  1  eye 
tracker  information  was  not  analyzed  in  more  depth. 

1.3  AOIs  2  through  6 

AOIs  2-6  are  informational  and  task  management  in  nature,  meaning  participants  did  not  need  to 
actively  monitor  these  AOIs  for  information  but  instead  look  to  them  when  directed  by  either 
incoming  message  notifications  or  their  current  task.  As  a  result,  total  fixations  and  glances  for 
AOIs  2-6  are  relatively  low  in  number,  even  though  collectively  these  AOIs  comprise  17.7%  of 
the  total  screen  area. 

1.4  AOI  2  Message  Center 

AOI  2  was  the  incoming  message  center,  where  the  participants  received  instruction  or  intelligence 
reports  from  either  mission  command  or  RoboFeader.  Incoming  messages  notify  the  participant  of 
changes  to  be  made  to  the  convoy  route,  and  are  accompanied  by  both  an  audible  signal  and  visual 
signals  (the  Replan  button  in  AOI  3  begins  flashing  yellow  and  the  affected  area  highlights  in  AOI 
1  Map).  Each  mission  had  six  route  changes,  and  according  to  table  3,  AOI  2  averaged  only  four 
Glances  in  the  Manual  and  Semi-Autonomous  conditions,  while  it  averaged  nine  Glances  in  the 
Fully  Autonomous  condition.  This  indicates  that  when  workload  levels  were  high,  participants 
did  not  read  the  incoming  intell  message,  but  instead  relied  upon  the  audio  prompts  and  color- 
coded  signals  that  appeared  on  the  Map  to  review  route  change  requests. 
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Repeated  measures  analysis  of  variance  (ANOVA)  showed  there  was  a  significant  effect  of 
Mission  Automation  Condition  on  Fixation  Count  in  AOI  2,  Wilks’  X  =  0.653,  F  (2,  28)  =  7.447, 
p  =  0.003,  partial  if  =  0.347  (figure  1-15).  There  were  significantly  more  fixations  in  AOI  2  in 
the  Fully  Autonomous  condition  (M=  16.933)  than  in  either  the  Manual  (M=  6.033;  AM  = 

10.900 ,p  =  0.001)  or  the  Semi-Autonomous  Conditions  (M=  8.467;  AM=  8.467, p  =  0.006). 
Fixation  Count  in  AOI  2  appeared  to  be  inversely  related  to  Perceived  Workload,  whereas  in 
high  workload  conditions  (Manual  and  Semi-Autonomous)  the  participant  used  AOI  2  only  when 
directed,  but  in  the  reduced  workload  condition  (Fully  Autonomous)  the  participant  did  not  wait 
for  system  notification  but  instead  adopted  a  more  proactive  monitoring  behavior. 


AOI  2  Fixation  Count  by 
Mission  Automation  Level 
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Manual  Semi  Full 


Figure  1-1.  AOI  2:  fixation  count  by  LOA. 

1.5  AOI  3  Replan  -  Accept/Reject  -  Continue 

AOI  3  was  used  to  begin  and  end  the  route  replan  activity,  as  well  as  to  accept  or  reject 
RoboLeader  suggested  route  changes  during  the  Fully  Autonomous  Mission  Condition.  All 
participants  used  this  AOI  as  part  of  their  route  management  tasks.  There  was  no  information  to 
be  gained  by  the  participant  by  monitoring  this  AOI,  and  the  buttons  in  this  AOI  were  only  active 
during  route  management.  For  these  reasons,  AOI  3  usage  data  were  not  analyzed  in  more  depth. 

1.6  AOI  4  Edit  Route  Dialogue 

AOI  4  was  used  to  make  the  modifications  to  the  convoy  route,  as  well  as  pause/resume 
individual  vehicle  movement  for  the  vehicle  spacing  task.  The  edit  route  buttons  were  activated 
by  clicking  the  Replan  button  in  AOI  3,  and  were  deactivated  by  clicking  the  Continue  button  in 
AOI  3.  The  Pause/Resume  button  was  active  during  the  Manual  Condition  when  the  participant 
was  responsible  for  maintaining  vehicle  spacing  of  the  convoy,  but  not  during  either  RoboLeader 
condition.  The  participant  would  gain  no  information  by  monitoring  this  AOI,  as  no  incoming 
information  or  notices  appeared  in  this  AOI.  For  these  reasons,  AOI  4  usage  data  were  not 
analyzed  in  more  depth. 
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1.7  AOI  5  Start  -  Undo  -  Threat 

AOI  5  contained  the  Start,  Undo,  and  Threat  buttons.  The  Start  button  is  used  to  begin  the 
mission  and  is  unused  afterward.  The  Threat  button  would  flash  yellow  as  a  visual  confirmation 
whenever  the  participant  clicked  on  a  threat,  however  clicking  on  the  Threat  button  had  no 
effect.  The  Undo  button  was  used  if  the  participant  felt  they  had  identified  a  non-threat  as  a 
threat.  The  participant  could  immediately  click  the  Undo  button  to  note  their  mistake,  and  it 
would  only  remove  the  mouse  click  immediately  preceding  the  Undo  click  from  the  data. 
Clicking  the  Undo  button  did  not  remove  the  threat  icon  from  the  Map.  The  participant  would 
gain  no  information  by  monitoring  this  AOI,  as  no  incoming  information  or  notices  appeared  in 
this  AOI.  For  these  reasons,  AOI  5  usage  data  were  not  analyzed  in  more  depth. 

1.8  AOI  6  Vehicle  List 

AOI  6  showed  the  list  of  vehicles,  where  the  experimenter  assigned  which  camera  feed  would 
appear  in  which  AOI.  Once  the  camera  feeds  were  properly  loaded,  there  would  be  no  need  for 
either  the  experimenter  or  operator  to  interact  further  with  this  AOI  during  the  mission.  For  this 
reason,  AOI  6  usage  data  were  not  analyzed  in  more  depth. 

1.9  AOI  7  UAS  Camera  Feed 

AOI  7  contained  the  UAS  camera  feed,  which  the  participant  was  required  to  monitor  to 
maintain  SA  of  the  surrounding  areas.  No  target  detection  tasks  were  associated  with  this  feed, 
nor  could  the  view  be  manipulated  by  the  participant,  so  it  was  not  expected  that  participants 
would  click  in  this  AOI.  It  was  expected  that  the  participants  with  the  highest  SA  scores  would 
have  the  most  fixations  and  longest  dwell  times  among  users  in  this  AOI.  Simple  Linear 
Regression  analysis  indicated  Glance  and  Fixation  behavior  in  AOI  7  was  not  predictive  of  SA 
scores  (table  1-1). 


Table  1-1.  AOI  7  usage:  eye  movement  relation  to  SA  scores. 


Manual 

Semi 

Full 

Beta 

t 

Sig. 

Beta 

t 

Sig. 

Beta 

t 

Sig. 

Glance  Count 

-0.957 

-0.689 

0.498 

-1.016 

-0.803 

0.431 

-1.533 

-1.097 

0.286 

Avg  Glance  Duration 

-0.090 

-0.226 

0.823 

-0.362 

-0.790 

0.438 

-0.445 

-1.086 

0.291 

Fixation  Count 

2.503 

0.834 

0.413 

0.240 

0.090 

0.929 

2.711 

0.689 

0.499 

Avg  Fixation  Duration 

0.451 

0.994 

0.331 

0.115 

0.276 

0.785 

-0.035 

-0.084 

0.934 

1.9.1  LOA 


There  was  a  significant  main  effect  of  LOA  in  AOI  7  on  both  Fixation  Count,  Wilks’  k  =  0.590, 
F  (2,  27)  =  9.395,  p  =  0.001,  partial  tf  =  0.410,  and  Glance  Count,  Wilks’  k  =  0.618,  F  (2,  27)  = 
8.345, p  =  0.002,  partial  rj  =  0.382.  The  Manual  Condition  had  significantly  fewer  Fixations  and 
Glances  than  either  RoboLeader  condition  (figure  1-1). 
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Figure  1-2.  AOI  7  fixations  (a)  and  glances  (b)  across  LOAs. 

1.9.2  Individual  Difference  Factors 

The  individual  difference  factor  Gaming  Experience  was  positively  correlated  with  Fixation 
Count  and  Glance  Count  in  AOI  7  in  the  RoboLeader  conditions  (table  1-2),  but  not  SpAO,  PAC, 
or  SpAC,  regardless  of  condition.  Average  Fixation  Duration  and  Average  Glance  Duration  were 
not  significantly  correlated  with  any  individual  difference  measures  in  AOI  7. 

Table  1-2.  AOI  7  usage:  gaming  experience  correlations  with  fixation 
and  glance  behavior,  by  LOA. 


Gaming  Experience 

Glances 

Count 

Avg 

Glance 

Duration 

Fix 

Count 

Avg 

Fixation 

Duration 

Manual 

Pearson’s  r 

0.214 

-0.118 

0.252 

-0.038 

Sig.  (2-tailed) 

0.255 

0.534 

0.179 

0.843 

Semi 

Pearson’s  r 

0.400a 

0.123 

0.386“ 

0.107 

Sig.  (2-tailed) 

0.028 

0.525 

0.035 

0.581 

Full 

Pearson’s  r 

0.4 1 5a 

0.029 

0.402a 

-0.099 

Sig.  (2-tailed) 

0.023 

0.884 

0.028 

0.623 

Correlation  is  significant  at  the  0.05  level  (2-tailed). 


There  was  no  significant  interaction  between  Gaming  Experience  and  EOA  on  Fixation  Count, 
Wilks’  X  =  0.881,  F  (2,  27)  =  1 .830,  y?  =  0.180,  partial  rj2  =  0.1 19,  nor  between  Gaming 
Experience  and  EOA  on  Glance  Count,  Wilks’  X  =  0.870,  F  (2,  27)  =  2.022,  p  =  0.152,  partial  rj2 
=  0.152. 

There  was  a  significant  between-subjects  effect  of  Gaming  Experience  on  Fixation  Count,  F  (1,  28)  = 
5.413,/?  =  0.027,  partial  rj2  =  0.162  (figure  I-3a),  and  Glance  Count,  F(  1,  28)  =  5.419,/?  =  0.027, 
partial ;/  =0.162  (figure  I-3b).  Frequent  Action  Gamers  had  made  significantly  more  Fixations 
and  Glances  in  AOI  7  than  All  Other  Gamers. 
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Figure  1-3.  AOI  7  fixations  (a)  and  glances  (b)  by  gaming  experience,  across  LOAs. 

1.10  AOI  8  UGV  Camera  Feed 

AOI  8  contained  the  UGV  camera  feed,  which  gave  the  participant  an  advanced  view  of  what  the 
convoy  would  be  encountering  ahead.  This  view  could  not  be  manipulated  by  the  participant, 
however,  this  AOI  could  be  used  in  the  target  detection  task  to  mark  threats  by  clicking  on  them, 
so  it  was  expected  that  individuals  who  scored  well  on  the  target  detection  task  and  had  the 
fewest  FAs  would  utilize  this  AOI  towards  that  end.  Simple  Linear  Regression  analysis  indicated 
Average  Fixation  Duration  was  predictive  of  Target  Detection  Scores  in  the  Fully  Autonomous 
condition  (table  1-3),  and  Glance  and  Fixation  behavior  in  AOI  8  was  not  predictive  of  FAs, 
regardless  of  condition  (table  1-4). 

Table  1-3.  AOI  8  fixation  and  glance  behavior  for  target  detection  task,  across  LOAs. 


Manual 

Semi 

Full 

Beta 

t 

Sig. 

Beta 

t 

Sig. 

Beta 

t 

Sig. 

Glance  Count 

-2.267 

-1.373 

0.183 

-0.360 

-0.163 

0.872 

-0.709 

-0.902 

0.377 

Avg  Glance  Duration 

-0.734 

-1.392 

0.178 

-0.108 

-0.151 

0.882 

0.667 

1.594 

0.126 

Fixation  Count 

4.468 

1.146 

0.264 

0.994 

0.320 

0.752 

1.436 

0.992 

0.332 

Avg  Fixation  Duration 

0.742 

1.470 

0.156 

0.128 

0.213 

0.834 

-0.980 

-2.323 

0.030 
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Table  1-4.  AOI  8  fixation  and  glance  behavior  for  FAs,  across  LOAs. 


Manual 

Semi 

Full 

Beta 

t 

Sig. 

Beta 

t 

Sig. 

Beta 

t 

Sig. 

Glance  Count 

-1.665 

-1.064 

0.299 

2.849 

1.489 

0.151 

1.431 

1.564 

0.133 

Avg  Glance  Duration 

-0.634 

-1.268 

0.218 

0.271 

0.435 

0.668 

0.444 

0.912 

0.372 

Fixation  Count 

2.333 

0.632 

0.534 

-4.289 

-1.592 

0.126 

-0.896 

-0.532 

0.600 

Avg  Fixation  Duration 

0.122 

0.256 

0.801 

-0.467 

-0.896 

0.380 

-0.113 

-0.229 

0.821 

1.10.1  LOA 

There  was  a  significant  main  effect  of  LOA  in  AOI  8  on  both  Fixation  Count,  Wilks’  X  =  0.523, 
F  ( 2,  28)  =  12.77 1 ,  p  <  0.001,  partial  if  =  0.477,  and  Glance  Count,  Wilks’  X  =  0.540,  F  ( 2,  28) 
=  1 1.902,  p  <  0.001,  partial  rj2  —  0.460.  The  Manual  Condition  had  significantly  fewer  Fixations 
and  Glances  than  either  RoboLeader  condition  (figure  1-4). 


Figure  1-4.  AOI  8  fixations  (a)  and  glances  (b)  across  LOAs. 


1.10.2  Individual  Difference  Factors 

PAC  was  not  correlated  with  Fixation  and  Glance  behavior  in  AOI  8.  SpAC  was  positively 
correlated  with  Average  Glance  Duration  in  AOI  8  in  the  Manual  Condition  only,  r  =  0.403, 
p  =  0.027,  2-tailed.  Glance  Count  and  Fixation  Count  in  AOI  8  were  moderately  correlated  with 
SpAO  in  the  RoboLeader  conditions  (table  1-5),  but  not  the  Manual  condition.  Glance  Count  and 
Fixation  Count  were  positively  correlated  with  Gaming  Experience  in  the  RoboLeader 
conditions  (table  1-6),  but  not  the  Manual  condition. 
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Table  1-5.  AOI  8  usage:  SpAO  correlations  with  fixation  and  glance  behavior,  by  LOA. 


Spatial  Ability  Orientation  (SpAO) 


Glances 

Count 

Avg 

Glance 

Duration 

Fix 

Count 

Avg 

Fixation 

Duration 

Manual 

Pearson’s  r 

0.225 

-0.068 

0.208 

-0.126 

Sig.  (2-tailed) 

0.233 

0.723 

0.271 

0.506 

Semi 

Pearson’s  r 

0.346 

0.079 

0.335 

0.074 

Sig.  (2-tailed) 

0.061 

0.678 

0.071 

0.699 

Full 

Pearson’s  r 

0.373“ 

-0.019 

0.357 

-0.177 

Sig.  (2-tailed) 

0.042 

0.922 

0.053 

0.357 

“  Correlation  is  significant  at  the  0.05  level  (2-tailed). 


Table  1-6.  AOI  8  usage:  gaming  experience  correlations  with  fixation  and 
glance  behavior,  by  LOA. 


Gaming  Experience 

Glances 

Count 

Avg 

Glance 

Duration 

Fix 

Count 

Avg 

Fixation 

Duration 

Manual 

Pearson’s  r 

0.228 

-0.035 

0.234 

-0.112 

Sig.  (2-tailed) 

0.225 

0.855 

0.213 

0.557 

Semi 

Pearson’s  r 

0.395“ 

0.057 

0.403“ 

0.027 

Sig.  (2-tailed) 

0.031 

0.763 

0.027 

0.889 

Full 

Pearson’s  r 

0.393“ 

0.003 

0.407“ 

-0.140 

Sig.  (2-tailed) 

0.032 

0.987 

0.026 

0.468 

“Correlation  is  significant  at  the  0.05  level  (2-tailed). 


1.10.3  SpAC 

There  was  a  marginally  significant  interaction  between  SpAC  and  LOA  on  Average  Glance 
Duration,  Wilks’  X  =  0.822,  F  (2,  26)  =  2.813 ,p  =  0.078,  partial  if  =  0.178.  In  the  Manual 
Condition,  participants  who  scored  higher  on  the  Spatial  Ability  Cube  Comparison  test  made,  on 
average,  longer  duration  glances  into  AOI  8  than  those  who  scored  lower  (figure  1-5),  but  this 
difference  was  not  significant  for  the  RoboLeader  conditions. 
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AOI  8  Avg  Glance  Duration  by  SpAC 


SOO 


Manual  Semi  Tull 


Figure  1-5.  AOI  8  average  glance  duration  by  SpAC, 
across  LOAs. 


1.10.4  SpAO 

There  was  no  significant  interaction  between  SpAO  and  LOA  on  Fixation  Count,  Wilks’  X  =  0.863, 
F  (2,  27)  =  2.145,/?  =  0.137,  partial  if  =  0.137,  nor  between  SpAO  and  LOA  on  Glance  Count, 
Wilks’  X  =  0.876,  F  (2,  27)  =  1.907,/?  =  0.168,  partial  q  =  0.124. 

There  was  a  marginally  significant  between-subjects  effect  of  SpAO  on  Fixation  Count,  F  (1,  28) 
=  3.462,/?  =  0.073,  partial  rj2  =  0.1 10,  and  Glance  Count,  F(l,  28)  =  3.740,/?  =  0.063, 
partial  rj  =  0.1 18  (figure  1-6).  Participants  with  High  SpAO  made  more  Fixations  and  Glances  in 
AOI  8  than  those  with  Low  SpAO,  across  all  LOA. 


Figure  1-6.  AOI  8  fixations  (a)  and  glances  (b)  by  SpAO,  across  LOAs. 

1.10.5  Gaming  Experience 

There  was  no  significant  interaction  between  Gaming  Experience  and  LOA  on  Fixation  Count, 
Wilks’  X  =  0.807,  F  (2,  27)  =  3.237,  p  =  0.055,  partial  if  =  0.193,  or  between  Gaming 
Experience  and  LOA  on  Glance  Count,  Wilks’  X  =  0.831,  F  (2,  27)  =  2.752,/?  =  0.082,  partial  rj2 
=  0.169. 
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There  was  a  significant  between-subjects  effect  of  Gaming  Experience  on  Fixation  Count, 

F(  1,  28)  =  4.908,  p  =  0.035,  partial  rj2  =  0.149,  and  Glance  Count,  F  (l,  28)  =  4.5149 ,p  =  0.043, 
partial ;/  =0.139  (figure  1-7).  Frequent  Action  Gamers  had  made  significantly  more  fixations 
and  glances  in  AOI  8  than  All  Other  Gamers,  however,  the  Average  Duration  of  their  fixations 
and  glances  were  not  different  from  All  Other  Gamers. 


AOI  8  Glance  Count  by  Gaming 


Experience 


dte! 


(b) 


AOI  8  Fixation  Count  by  Gaming 
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Figure  1-7.  AOI  8  fixations  (a)  and  glances  (b)  by  gaming  experience,  across  LOAs. 

1.10.6  Clicking  Behavior 

Gaming  Experience  was  also  positively  correlated  with  whether  the  participant  clicked  in  the 
AOI  in  the  Fully  Autonomous  condition,  r  =  0.38 1,/?  =  0.038,  2-tailed,  and  moderately 
correlated  in  the  Manual  condition,  r  =  0.321  ,p  =  0.083,  2-tailed,  but  not  correlated  in  the  Semi- 
Autonomous  condition,  r  =  0.247,  p  =  0.188,  2-tailed.  Chi-Squared  statistical  analysis  (using 
Yates’  Correction  for  Continuity)  of  the  likelihood  of  Gaming  Experience  affecting  Clicking 
Behavior  in  AOI  8  indicated  the  groups  were  not  significantly  different,  (Manual:  X2  (1 , 30)  = 
1.926,/?  =  0.165;  Semi:X2(l,  30)  =  0.956,/?  =  0.328;  Fully:  X2(l,  30)  =  2.868,/?  =  0.090).  It  is 
interesting  to  note  that  among  the  Frequent  Action  Gamers  the  number  of  participants  that  did  or 
did  not  click  in  the  AOI  was  roughly  evenly  split  in  each  condition,  while  the  number  of  All 
Other  Gamers  who  did  not  click  in  AOI  8  was  consistently  higher  in  every  mission  condition 
(figure  1-8).  Target  Detection  performance  was  not  correlated  with  clicking  behavior  in  any 
Mission  Condition. 
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Clicking  Behavior  in  AOI  8,  by  Participant  Gaming 
Experience 
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Figure  1-8.  Percent  of  participants  that  clicked  in  AOI  8,  sorted  by  gaming 
experience,  across  LOAs. 

1.11  AOIs  9, 10,  and  11:  MGV  Forward  180°  Camera  Feed 

AOIs  9,  10  and  1 1  comprise  the  MGV  forward  180°  camera  feed,  and  collectively  account  for 
21.3%  of  the  screen  area.  Objects  appear  on  the  horizon  in  AOI  10  (straight  ahead),  then  as  the 
vehicles  approach  the  object  moves  from  AOI  10  into  either  AOI  9  (on  the  right)  or  1 1  (on  the 
left)  before  disappearing  from  the  forward  camera  feeds.  The  views  in  these  AOIs  cannot  be 
manipulated  by  the  participant,  and  monitoring  these  feeds  was  essential  for  identifying  threats 
for  the  target  detection  task,  as  well  as  maintaining  SA.  It  was  expected  that  AOI  10  will  have 
more  fixations  and  dwells  than  either  9  or  11,  primarily  due  to  its  central  location  and  proximity 
to  AOI  1 .  It  was  also  expected  that  participants  who  utilized  all  three  AOIs  for  the  target 
detection  task  will  have  higher  scores  and  fewer  FAs  than  those  who  did  not. 

1.12  AOI  9  MGV  Forward  Right 

Simple  Linear  Regression  analysis  indicated  that  Fixation  and  Glance  behavior  in  AOI  9  was  not 
predictive  of  performance  on  the  Target  Detection  Task  (table  1-7)  or  of  FAs  (table  1-8), 
regardless  of  mission  condition. 

Table  1-7.  AOI  9  fixation  and  glance  behavior  for  target  detection  across  LOAs. 


Manual 

Semi 

Full 

Beta 

t 

Sig. 

Beta 

t 

Sig. 

Beta 

t 

Sig. 

Glance  Count 

1.875 

0.873 

0.392 

-0.420 

-0.321 

0.751 

-1.637 

-1.088 

0.288 

Avg  Glance  Duration 

1.158 

0.871 

0.393 

-1.038 

-1.365 

0.186 

-1.320 

-1.274 

0.216 

Fixation  Count 

-1.150 

-0.410 

0.686 

-1.204 

-0.610 

0.548 

2.774 

1.596 

0.125 

Avg  Fixation  Duration 

-1.239 

-0.951 

0.352 

0.038 

0.056 

0.955 

0.355 

0.554 

0.585 
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Table  1-8.  AOI  9  fixation  and  glance  behavior  for  FAs  across  LOAs. 


Manual 

Semi 

Full 

Beta 

t 

Sig. 

Beta 

t 

Sig. 

Beta 

t 

Sig. 

Glance  Count 

-3.423 

-1.666 

0.110 

0.186 

0.119 

0.906 

-1.002 

-0.603 

0.553 

Avg  Glance  Duration 

-2.226 

-1.749 

0.094 

-0.058 

-0.064 

0.949 

-0.681 

-0.595 

0.558 

Fixation  Count 

2.713 

1.011 

0.323 

-0.620 

-0.265 

0.794 

0.773 

0.402 

0.691 

Avg  Fixation  Duration 

1.794 

1.439 

0.164 

-0.270 

-0.340 

0.737 

-0.057 

-0.081 

0.936 

1.12.1  LOA 

There  was  a  significant  main  effect  of  LOA  on  Fixation  Count,  Wilks’  X  =  0.371,  F  (2,  28)  =  23.781, 
p  <  0.001,  partial  if  =  0.629  (figure  I-9a),  and  Glance  Count,  Wilks’  A.  =  0.357,  F  (2,  28)  =  25. 193, 
p  <  0.001,  partial  if  =  0.643  (figure  I-9b).  There  were  significantly  fewer  Fixations  and  Glances 
in  AOI  9  in  the  Manual  condition  than  in  the  RoboLeader  conditions. 


Figure  1-9.  AOI  9  fixation  count  (a)  and  glance  count  (b)  across  LOAs. 
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There  was  a  significant  main  effect  of  LOA  on  Average  Fixation  Duration,  Wilks’  X  =  0.161 ,  F 
(2,  28)  =  4.244 ,p  =  0.025,  partial  rj2  =  0.233  (figure  10a),  and  Average  Glance  Duration,  Wilks’ 
X  =  0.546,  F (2,  28)  =  1 1.654 ,p<  0.001,  partial  rj2  =  0.454  (figure  10b).  Fixations  and  Glances 
in  AOI  9  were  significantly  shorter  in  the  Manual  condition  than  in  the  RoboLeader  conditions. 


Figure  I- 1 0.  AOI  9  average  fixation  duration  (a)  and  average  glance  duration  (b)  across  LOAs. 


1.12.2  Individual  Difference  Factors 

Average  Glance  Duration  and  Average  Fixation  Duration  were  moderately  negatively  correlated 
with  PAC  scores  (table  1-9)  in  the  Fully  Autonomous  condition,  but  not  in  the  Manual  or  Semi- 
Autonomous  conditions.  Glance  Count  and  Fixation  Count  in  AOI  9  were  negatively  correlated 
with  SpAC  scores  in  the  Semi -Autonomous  condition  (table  I- 10),  but  not  the  Manual  or  Fully 
Autonomous  conditions.  Average  Glance  Duration  in  AOI  9  was  negatively  correlated  with 
SpAO  scores  (table  1-1 1)  in  the  Manual  and  Fully  Autonomous  conditions,  but  not  in  the  Semi- 
Autonomous  condition.  Gaming  Experience  was  not  correlated  with  Fixation  or  Glance  behavior 
in  AOI  9. 


Table  1-9.  AOI  9  usage:  PAC  correlations  with  fixation  and  glance  behavior 
by  LOA. 


Attentional  Control  (PAC) 

Glances 

Count 

Avg 

Glance 

Duration 

Fix 

Count 

Avg 

Fixation 

Duration 

Manual 

Pearson’s  r 

-0.025 

0.223 

0.012 

0.140 

Sig.  (2-tailed) 

0.898 

0.236 

0.949 

0.462 

Semi 

Pearson’s  r 

0.144 

0.260 

0.218 

0.070 

Sig.  (2-tailed) 

0.448 

0.166 

0.247 

0.714 

Full 

Pearson’s  r 

0.139 

0.365a 

0.186 

0.360 

Sig.  (2-tailed) 

0.464 

0.047 

0.325 

0.051 

Correlation  is  significant  at  the  0.05  level  (2-tailed). 
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Table  I- 10.  AOI  9  usage:  SpAC  correlations  with  fixation  and  glance  behavior  by 
LOA. 


Spatial  Ability  Cube  Comparison  (SpAC) 

Glances 

Count 

Avg 

Glance 

Duration 

Fix  Count 

Avg 

Fixation 

Duration 

Manual 

Pearson’s  r 

-0.054 

0.319 

-0.022 

0.324 

Sig.  (2-tailed) 

0.776 

0.086 

0.907 

0.081 

Semi 

Pearson’s  r 

-0.450a 

0.062 

-0.4 17a 

0.212 

Sig.  (2-tailed) 

0.013 

0.746 

0.022 

0.261 

Full 

Pearson’s  r 

-0.286 

-0.037 

-0.285 

0.021 

Sig.  (2-tailed) 

0.125 

0.848 

0.127 

0.914 

a Correlation  is  significant  at  the  0.05  level  (2-tailed). 


Table  1-1 1.  AOI  9  usage:  SpAO  correlations  with  fixation  and  glance  behavior 
by  LOA. 


Spatial  Ability  Orientation  (SpAO) 


Glances 

Count 

Avg 

Glance 

Duration 

Fix  Count 

Avg 

Fixation 

Duration 

Manual 

Pearson’s  r 

-0.021 

-0.390a 

-0.134 

-0.211 

Sig.  (2-tailed) 

0.911 

0.033 

0.479 

0.264 

Semi 

Pearson’s  r 

-0.001 

-0.183 

-0.096 

-0.009 

Sig.  (2-tailed) 

0.997 

0.334 

0.614 

0.963 

Full 

Pearson’s  r 

-0.164 

-0.468b 

-0.301 

-0.277 

Sig.  (2-tailed) 

0.388 

0.009 

0.106 

0.138 

a  Correlation  is  significant  at  the  0.05  level  (2-tailed). 
b  Correlation  is  significant  at  the  0.01  level  (2-tailed). 


1.12.3  PAC 

There  was  no  significant  interaction  between  PAC  and  LOA  on  Average  Fixation  Duration, 
Wilks’  X  =  0.900,  F  (2,  27)  =  1.493 ,p  =  0.243,  partial  if  =  0.100,  or  between  PAC  and  LOA  on 
Average  Glance  Duration,  Wilks’  X  =  0.945,  F  (2,  27)  =  0.787,  p  =  0.465,  partial  if  =  0.055. 

There  was  no  significant  between-subjects  effect  of  PAC  on  Average  Fixation  Duration  in  AOI 
9,  F  (1,  28)  =  1.600,  p  =  0.216,  partial  if  =  0.054  (figure  1-1  la),  however,  there  was  a  marginally 
significant  between-subjects  effect  on  Average  Glance  Duration,  F  (1,  28)  =  3.721,/)  =  0.064, 
partial  if  =  0.1 17  (figure  1-1  lb).  Participants  with  higher  PAC  scores  had  longer  Glances  in  AOI 
9  than  those  with  low  PAC  scores,  across  all  mission  conditions. 
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Figure  1-1 1.  AOI  9  average  fixation  duration  (a)  and  average  glance  duration  (b)  by  PAC,  across  LOAs. 

1.12.4  SpAC 

There  was  a  significant  interaction  between  SpAC  and  LOA  on  Fixation  Count,  Wilks’  X  = 
0.761,  F  (2,  27)  =  4.233 ,p  =  0.025,  partial  if  =  0.239  (figure  I- 12a),  as  well  as  between  SpAC 
and  LOA  on  Glance  Count,  Wilks’  X  =  0.751,  F  (2,  27)  =  4.483, p  =  0.021,  partial  if  =  0.249 
(figure  I- 12b).  All  participants  had  similar  numbers  of  fixations  and  glances  in  AOI  9  during  the 
Manual  condition,  however,  participants  who  scored  low  on  the  Spatial  Ability  Cube 
Comparison  Test  had  significantly  more  fixations  and  glances  in  AOI  9  in  the  RoboLeader 
conditions  than  those  who  scored  high  on  the  SpAC. 
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Figure  1-12.  AOI  9  fixations  (a)  and  glances  (b)  by  SpAC,  across  LOAs. 


1.12.5  SpAO 

There  was  no  significant  interaction  between  SpAO  and  LOA  on  Average  Fixation  Duration, 
Wilks’  X  =  0.916,  F  (2,  27)  =  1.231,/?  =  0.308,  partial  rj2  =  0.084,  or  between  SpAO  and  LOA  on 
Average  Glance  Duration,  Wilks’  X=  0.849,  F  ( 2,  27)  =  2.396,  p  =  0.1 10,  partial  if  =  0.151. 
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There  was  no  significant  between-subjects  effect  of  SpAO  on  Average  Fixation  Duration,  F 
(1,  28)  =  1.216,/?  =  0.279,  partial  if  =  0.042  (figure  I- 13a),  however  there  was  a  significant 
between-subjects  effect  of  SpAO  on  Average  Glance  Duration,  F(l,  28)  =  6.189,/?  =  0.019, 
partial  rj  =0.181  (figure  I- 13b).  Participants  low  in  SpAO  had  longer  Glance  Durations  in  AOI 
9  than  those  with  higher  SpAO  scores. 
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Figure  1-13.  AOI  9  average  fixation  duration  (a)  and  average  glance  duration  (b)  by  SpAO,  across  LOAs. 


1.12.6  Clicking  Behavior 

Clicking  Behavior  in  AOI  9  was  not  significantly  correlated  with  performance  on  Target 
Detection  Task  or  number  of  FAs,  nor  was  it  correlated  with  any  individual  difference  measures. 
Overall,  81%  of  participants  did  click  in  AOI  9,  with  this  percentage  being  lowest  in  the  Manual 
condition  (76.7%)  and  increasing  as  the  level  of  automation  assistance  increased,  Semi  (80.0%), 
to  the  highest  in  the  Fully  Autonomous  condition  (86.7%),  (figure  1-14). 


Figure  1-14.  Percent  of  participants  that  clicked  in  AOI  9  at  least  once,  across 
LOAs. 
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1.13  AOI 10  MGV  Forward  Center 


Simple  Linear  Regression  analysis  indicated  that  Fixation  and  Glance  behavior  in  AOI  10  was 
not  predictive  of  performance  on  the  Target  Detection  Task  (table  1-12)  or  of  FAs  (table  1-13), 
regardless  of  mission  condition. 

Table  1-12.  AOI  10  fixation  and  glance  behavior  for  target  detection,  across  LOAs. 


Manual 

Semi 

Full 

Beta 

t 

Sig. 

Beta 

t 

Sig. 

Beta 

t 

Sig. 

Glance  Count 

0.224 

0.185 

0.855 

0.395 

0.400 

0.693 

-1.258 

-1.828 

0.081 

Avg  Glance  Duration 

0.348 

0.337 

0.739 

-0.348 

-0.364 

0.719 

-1.569 

-1.988 

0.059 

Fixation  Count 

1.831 

0.835 

0.413 

-1.622 

-0.852 

0.403 

2.662 

1.820 

0.082 

Avg  Fixation  Duration 

0.531 

0.670 

0.510 

-0.686 

-0.686 

0.500 

1.012 

1.269 

0.218 

Table  1-13.  AOI  10  fixation  and  glance  behavior  for  FAs  across  LOAs. 


Manual 

Semi 

Full 

Beta 

t 

Sig. 

Beta 

t 

Sig. 

Beta 

t 

Sig. 

Glance  Count 

1.892 

1.762 

0.092 

-0.753 

-0.686 

0.500 

-0.841 

-1.092 

0.287 

Avg  Glance  Duration 

1.500 

1.642 

0.115 

-0.499 

-0.470 

0.643 

-0.078 

-0.088 

0.930 

Fixation  Count 

-1.124 

-0.579 

0.568 

-0.428 

-0.203 

0.841 

-0.466 

-0.285 

0.779 

Avg  Fixation  Duration 

0.317 

0.452 

0.655 

-0.284 

-0.256 

0.800 

-0.855 

-0.956 

0.349 

1.13.1  LOA 

There  was  a  significant  main  effect  of  LOA  on  Average  Glance  Duration,  Wilks’  X  =  0.721,  F 
(2,  28)  =  5.424,  p  =  0.010,  partial  >f  =  0.279  (figure  1-15).  Average  Glance  Duration  in  AOI  10 
was  lowest  in  the  Manual  condition,  highest  in  the  RoboLeader  conditions.  There  was  no 
significant  main  effect  of  LOA  on  Average  Fixation  Duration,  Wilks’  X  =  0.832,  F  (2,  28)  =  2.828, 
p  =  0.076,  partial  rf  =  0.168. 
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Figure  1-15.  AOI  10  average  glance  duration  across  LOAs. 

1.13.2  Individual  Difference  Factors 

PAC  was  not  correlated  with  Fixation  and  Glance  behavior  in  AOI  10.  Average  Glance  Duration 
and  Average  Fixation  Duration  in  AOI  10  were  moderately  correlated  with  SpAC  scores 
(table  1-14)  in  the  Manual  condition,  but  not  in  the  Semi-  or  Fully  Autonomous  conditions. 
Average  Glance  Duration  was  negatively  correlated  with  SpAO  scores  (table  1-15)  in  the  Fully 
Autonomous  conditions  but  not  in  the  Manual  or  Semi-Autonomous  conditions.  Average 
Fixation  Duration  in  AOI  10  is  negatively  correlated  with  Gaming  Experience  (table  1-16)  in  the 
Manual  condition,  but  not  in  the  Semi-  or  Fully  Autonomous  conditions. 

Table  1-14.  AOI  10  usage:  SpAC  correlations  with  fixation  and  glance  behavior  by 
LOA. 


Spatial  Ability  Cube  Comparison  (SpAC) 


Glances 

Count 

Total 

Glance 

Time 

Avg 

Glance 

Duration 

Fix 

Count 

Total 

Fixation 

Time 

Avg 

Fixation 

Duration 

Manual 

Pearson’s  r 

0.101 

0.347 

0.340 

0.276 

0.348 

0.316 

Sig.  (2-tailed) 

0.597 

0.060 

0.066 

0.140 

0.059 

0.089 

Semi 

Pearson’s  r 

0.033 

-0.210 

0.255 

-0.079 

0.043 

0.262 

Sig.  (2-tailed) 

0.862 

0.266 

0.173 

0.677 

0.823 

0.162 

Full 

Pearson’s  r 

0.057 

-0.181 

0.203 

-0.002 

0.048 

0.116 

Sig.  (2-tailed) 

0.765 

0.339 

0.282 

0.992 

0.800 

0.540 
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Table  1-15.  AOI  10  usage:  SpAO  correlations  with  fixation  and  glance  behavior  by  LOA. 


Spatial  Ability  Orientation  (SpAO) 

Glances 

Count 

Total 

Glance 

Time 

Avg 

Glance 

Duration 

Fix 

Count 

Total 

Fixation 

Time 

Avg 

Fixation 

Duration 

Manual 

Pearson’s  r 

0.160 

-0.013 

-0.147 

0.048 

-0.010 

-0.222 

Sig.  (2 -tailed) 

0.399 

0.945 

0.439 

0.801 

0.959 

0.239 

Semi 

Pearson’s  r 

-0.085 

0.061 

-0.233 

-0.061 

-0.089 

-0.078 

Sig.  (2 -tailed) 

0.654 

0.750 

0.216 

0.749 

0.639 

0.682 

Full 

Pearson’s  r 

-0.276 

-0.037 

-0.398a 

-0.248 

-0.261 

-0.163 

Sig.  (2 -tailed) 

0.140 

0.846 

0.029 

0.187 

0.164 

0.390 

a  Correlation  is  significant  at  the  0.05  level  (2-tailed). 


Table  1-16.  AOI  10  usage:  gaming  experience  correlations  with  fixation  and  glance 
behavior  by  LOA. 


Gaming  Experience 

Glances 

Count 

Total 

Glance 

Time 

Avg 

Glance 

Duration 

Fix 

Count 

Total 

Fixation 

Time 

Avg 

Fixation 

Duration 

Manual 

Pearson’s  r 

-0.167 

-0.222 

-0.057 

-0.127 

-0.238 

-0.381a 

Sig.  (2-tailed) 

0.377 

0.239 

0.763 

0.505 

0.204 

0.038 

Semi 

Pearson’s  r 

-0.068 

0.039 

-0.081 

0.042 

-0.090 

-0.199 

Sig.  (2-tailed) 

0.721 

0.837 

0.670 

0.824 

0.635 

0.291 

Full 

Pearson’s  r 

0.104 

0.166 

0.042 

0.189 

0.095 

-0.105 

Sig.  (2-tailed) 

0.583 

0.380 

0.826 

0.318 

0.619 

0.579 

Correlation  is  significant  at  the  0.05  level  (2-tailed). 


1.13.3  SpAC 

There  was  no  significant  interaction  between  SpAC  and  LOA  on  Average  Fixation  Duration, 
Wilks’  X  =  0.960,  F  (2,  27)  =  0.567,  p  =  0.574,  partial  rj2  =  0.040,  nor  Average  Glance  Duration, 
Wilks’  X  =  0.996,  F  (2,  27)  =  0.058,  p  =  0.944,  partial  if  =  0.004.  There  was  no  significant 
between-subjects  effect  of  SpAC  on  Average  Fixation  Duration,  F  ( 1 ,  28)  =  2.048,  p  =  0.163, 
partial  if  =  0.068,  or  Average  Glance  Duration,  F  ( 1 ,  28)  =  2.782,  p  =  0.106,  partial  if  =  0.090. 

1.13.4  SpAO 

There  was  no  significant  interaction  between  SpAO  and  LOA  on  Average  Glance  Duration, 
Wilks’  X  =  0.854,  F  (2,  27)  =  2.305,/?  =  0.1 19,  partial  if  =  0.146.  There  was  no  significant 
between-subjects  effect  of  SpAO  on  Average  Glance  Duration  in  AOI  10,  F(l,  28)  =  3.056,/?  = 
0.091,  partial  if  =  0.098. 
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1.13.5  Gaming  Experience 

There  was  no  significant  interaction  between  Gaming  Experience  and  LOA  on  Average  Fixation 
Duration,  Wilks’  X  =  0.939,  F  (2,  27)  =  0.880,/?  =  0.426,  partial  if  =  0.061.  There  was  no 
significant  between-subjects  effect  of  Gaming  Experience  on  Average  Fixation  Duration,  F  (1 , 
28)  =  1.939,/?  =  0.175,  partial  rj2  =  0.065. 


1.13.6  Clicking  Behavior 


Clicking  Behavior  in  AOI  10  was  correlated  with  performance  on  Target  Detection  Task  in  the 
Manual,  r  =  0.410,/?  =  0.024,  2-tailed,  and  Semi-Autonomous,  r  =  0.327,/?  =  0.077,  2-tailed, 
conditions,  but  not  in  the  Fully  Autonomous,  r  =  0.254,  /?  =  0.175,  ns,  condition.  Clicking 
Behavior  in  AOI  10  was  not  correlated  with  the  number  of  FAs  in  any  condition,  or  with  any 
individual  difference  measures.  Overall,  93%  of  participants  did  click  in  AOI  10,  with  this 
percentage  being  lowest  in  the  Manual  condition  (90%)  and  higher  in  the  RoboFeader 
conditions;  Semi  (96.7%),  and  Fully  Autonomous  condition  (93.3%),  (figure  1-16). 


Participants  (Overall)  that  Clicked  in 
AOI  10 


Figure  1-16.  Percent  of  participants  that  clicked  in  AOI  10  at  least  once,  across  LOAs. 

1.14  AOI  11  MGV  Forward  Left 

Simple  Linear  Regression  analysis  indicated  that  Fixation  and  Glance  behavior  in  AOI  1 1  was 
not  predictive  of  performance  on  the  Target  Detection  Task  (table  1-17)  or  of  FAs  (table  1-18), 
regardless  of  mission  condition. 
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Table  1-17.  AOI  1 1  fixation  and  glance  behavior  for  target  detection  across  LOAs. 


Manual 

Semi 

Full 

Beta 

t 

Sig. 

Beta 

t 

Sig. 

Beta 

t 

Sig. 

Glance  Count 

-0.375 

-0.219 

0.828 

0.591 

0.317 

0.754 

-1.061 

-1.348 

0.192 

Avg  Glance  Duration 

0.278 

0.279 

0.783 

0.078 

0.072 

0.943 

-0.726 

-1.305 

0.205 

Fixation  Count 

1.437 

0.611 

0.547 

-2.762 

-1.054 

0.303 

2.845 

1.645 

0.114 

Avg  Fixation  Duration 

0.051 

0.054 

0.957 

-1.099 

-0.984 

0.336 

0.195 

0.428 

0.673 

Table  1-18.  AOI  1 1  fixation  and  glance  behavior  for  FAs  across  LOAs. 


Manual 

Semi 

Full 

Beta 

t 

Sig. 

Beta 

t 

Sig. 

Beta 

t 

Sig. 

Glance  Count 

-0.114 

-0.073 

0.942 

1.548 

0.897 

0.380 

0.885 

0.990 

0.333 

Avg  Glance  Duration 

-0.402 

-0.443 

0.662 

0.776 

0.770 

0.450 

0.809 

1.280 

0.214 

Fixation  Count 

0.277 

0.129 

0.898 

4.034 

1.660 

0.111 

0.634 

0.323 

0.750 

Avg  Fixation  Duration 

0.730 

0.855 

0.402 

1.468 

1.417 

0.170 

-0.461 

-0.888 

0.384 

1.14.1  LOA 

There  was  no  significant  main  effect  of  LOA  on  Average  Fixation  Duration,  Wilks’  X  =  0.860,  F 
(2,  28)  =  2.274,  p  =  0.122,  partial  rf  =  0.140,  however  there  was  a  significant  main  effect  of 
LOA  on  Average  Glance  Duration,  Wilks’  X  =  0.683,  F  (2,  28)  =  6.499,  p  =  0.005,  partial  if  = 
0.3 17  (figure  1-17).  While  Average  Fixation  Duration  in  AOI  1 1  was  consistent  throughout  LOA, 
the  Average  Glance  Duration  in  both  of  the  RoboLeader  conditions  was  significantly  higher  than 
those  in  the  Manual  condition. 
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Figure  1-17.  AOI  11  average  glance  duration 
across  LOAs. 


There  was  a  significant  main  effect  of  LOA  on  Fixation  Count,  Wilks’  X  =  0.429,  F  (2,  28)  = 
18.6 12,  p  <  0.001,  partial  if  =  0.571  (figure  I- 18a)  and  Glance  Count,  Wilks’  X  =  0.438,  F  (2, 
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28)  =  \1.969,p<  0.001,  partial  if  =  0.562  (Figure  I- 18b).  Fixation  and  Glance  Counts  in  AOI 
1 1  was  significantly  lower  in  the  Manual  condition  than  for  the  RoboLeader  conditions,  across 
all  participants. 
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Figure  1-18.  AOI  11  average  fixation  duration  (a)  and  average  glance  duration  (b)  across  LOAs. 

1.14.2  Individual  Difference  Factors 

Fixation  Count  in  AOI  1 1  was  moderately  correlated  with  PAC  (table  1-19)  in  the  Manual 
condition,  but  not  correlated  in  the  Semi-Autonomous  and  Fully  Autonomous  conditions. 
Average  Fixation  Duration  and  Average  Glance  Duration  were  negatively  correlated  with  SpAC 
scores  (table  1-20)  in  the  Manual  condition,  but  not  correlated  in  the  Semi-Autonomous  and 
Fully  Autonomous  conditions.  Fixation  Count  was  also  moderately  correlated  with  AOI  1 1  usage 
in  the  Manual  condition  only.  Average  Glance  Duration  in  AOI  1 1  was  negatively  correlated 
with  SpAO  scores  (table  1-21)  in  the  Manual  condition,  but  not  correlated  in  the  Semi- 
Autonomous  and  Fully  Autonomous  conditions.  Average  Fixation  Duration  and  Average  Glance 
Duration  in  AOI  1 1  were  negatively  correlated  with  Gaming  Experience  (table  1-22)  in  the 
Manual  condition,  moderately  negatively  correlated  in  the  Semi-Autonomous  condition,  and  not 
correlated  in  the  Fully  Autonomous  condition. 
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Table  1-19.  AOI  1 1  usage:  PAC  correlations  with  fixation  and  glance  behavior  by  LOA. 


Attentional  Control 

Glances 

Count 

Total 

Glance 

Time 

Avg 

Glance 

Duration 

Fix 

Count 

Total 

Fixation 

Time 

Avg 

Fixation 

Duration 

Manual 

Pearson’s  r 

-0.256 

-0.254 

-0.165 

-0.317 

-0.233 

0.009 

Sig.  (2-tailed) 

0.172 

0.176 

0.383 

0.088 

0.215 

0.961 

Semi 

Pearson’s  r 

-0.072 

-0.066 

-0.052 

-0.094 

-0.061 

0.058 

Sig.  (2-tailed) 

0.707 

0.728 

0.787 

0.621 

0.748 

0.761 

Full 

Pearson’s  r 

0.035 

0.122 

0.260 

0.134 

0.106 

0.136 

Sig.  (2-tailed) 

0.853 

0.519 

0.165 

0.481 

0.576 

0.474 

Table  1-20.  AOI  1 1  usage:  SpAC  correlations  with  fixation  and  glance  behavior  by  LOA. 


Spatial  Ability  Cube  Comparison  (SpAC) 

Glances 

Count 

Total 

Glance 

Time 

Avg 

Glance 

Duration 

Fix 

Count 

Total 

Fixation 

Time 

Avg 

Fixation 

Duration 

Manual 

Pearson’s  r 

0.292 

0.4 13a 

0.533b 

0.357 

0.406a 

0.461a 

Sig.  (2 -tailed) 

0.118 

0.023 

0.002 

0.053 

0.026 

0.010 

Semi 

Pearson’s  r 

-0.101 

0.044 

0.246 

-0.046 

0.025 

0.146 

Sig.  (2 -tailed) 

0.597 

0.815 

0.190 

0.811 

0.897 

0.442 

Full 

Pearson’s  r 

0.023 

-0.027 

-0.275 

0.033 

-0.037 

-0.217 

Sig.  (2 -tailed) 

0.902 

0.887 

0.141 

0.862 

0.846 

0.249 

a  Correlation  is  significant  at  the  0.05  level  (2-tailed). 
b  Correlation  is  significant  at  the  0.01  level  (2-tailed). 
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Table  1-21.  AOI  1 1  usage:  SpAO  correlations  with  fixation  and  glance  behavior  by  LOA. 


Spatial  Ability  Orientation  (SpAO) 


Glances 

Count 

Total 

Glance 

Time 

Avg 

Glance 

Duration 

Fix 

Count 

Total 

Fixation 

Time 

Avg 

Fixation 

Duration 

Manual 

Pearson’s  r 

0.136 

-0.059 

-0.365a 

-0.035 

-0.060 

-0.111 

Sig.  (2-tailed) 

0.473 

0.756 

0.048 

0.854 

0.754 

0.559 

Semi 

Pearson’s  r 

0.100 

-0.009 

-0.177 

0.026 

-0.036 

-0.112 

Sig.  (2-tailed) 

0.601 

0.961 

0.349 

0.893 

0.849 

0.555 

Full 

Pearson’s  r 

0.035 

0.163 

0.148 

0.122 

0.151 

0.048 

Sig.  (2-tailed) 

0.852 

0.391 

0.436 

0.521 

0.425 

0.802 

Correlation  is  significant  at  the  0.05  level  (2-tailed). 


Table  1-22.  AOI  1 1  usage:  gaming  experience  correlations  with  fixation  and  glance  behavior  by  LOA. 


Gaming  Experience 

Glances 

Count 

Total 

Glance 

Time 

Avg 

Glance 

Duration 

Fix 

Count 

Total 

Fixati 

on 

Time 

Avg 

Fixation 

Duration 

Manual 

Pearson’s  r 

-0.053 

-0.173 

-0.374a 

-0.080 

-0.177 

-0.439a 

Sig.  (2-tailed) 

0.781 

0.361 

0.042 

0.674 

0.349 

0.015 

Semi 

Pearson’s  r 

0.160 

0.017 

-0.231 

0.130 

-0.020 

-0.312 

Sig.  (2-tailed) 

0.397 

0.929 

0.219 

0.495 

0.915 

0.093 

Full 

Pearson’s  r 

0.065 

-0.113 

-0.195 

-0.037 

-0.120 

-0.195 

Sig.  (2-tailed) 

0.734 

0.552 

0.302 

0.846 

0.529 

0.302 

a  Correlation  is  significant  at  the  0.05  level  (2-tailed). 


1.14.3  PAC 

There  was  no  significant  interaction  between  PAC  and  LOA  on  Fixation  Count,  Wilks’  X  = 

0.900,  F  (2,  27)  =  1 .504,  p  =  0.240,  partial  rj2  =  0. 100.  There  was  no  significant  between- 
subjects  effect  of  PAC  on  Fixation  Count,  F  (1,  28)  =  0.096,  p  =  0.759,  partial  rj2  =  0.003. 

1.14.4  SpAC 

There  was  a  marginally  significant  interaction  between  SpAC  and  LOA  on  Average  Fixation 
Duration,  Wilks’  X  =  0.802,  F  (2,  27)  =  3.335, p  =  0.051,  partial  rf  =  0.198  (figure  1-19). 
Participants  who  scored  higher  on  the  SpAC  test  had  longer  Fixations  across  all  mission 
conditions,  while  participants  who  scored  lower  on  the  SpAC  test  had  much  shorter  fixations  in 
AOI  1 1  during  the  Manual  mission  condition.  Low  SpAC  participants’  Fixation  Duration 
increased  to  near  that  of  the  High  SpAC  participants  in  the  RoboLeader  conditions.  There  was  no 
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significant  interaction  between  SpAC  and  LOA  on  Average  Glance  Duration,  Wilks’  k  =  0.885, 
F  (2,  27)  =  1 .748,  =  0.193,  partial  t f  =  0.1 15. 


Figure  1-19.  Average  fixation  duration  by  SpAC, 
across  LOAs. 

There  was  a  significant  between-subjects  effect  of  SpAC  on  Average  Glance  Duration,  F  (1,  28) 
=  4.408,/)  =  0.045,  partial  rj  =  0.136,  (figure  1-20).  Average  Glance  Duration  was  significantly 
shorter  for  low  SpAC  participants  in  AOI  1 1  than  for  high  SPAC  participants,  across  all  Mission 
conditions. 
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Figure  1-20.  AOI  1 1  average  glance  duration  by 
SpAC,  across  LOAs. 


1.14.5  SpAO 

There  was  no  significant  interaction  between  SpAO  and  LOA  on  Average  Glance  Duration, 
Wilks’  k  =  0.944,  F  (2,  27)  =  0.808,  p  =  0.456,  partial  t f  =  0.056. 

There  was  a  marginally  significant  between-subjects  effect  of  SpAO  on  Average  Glance 
Duration,  F { 1,  28)  =  3.628,/?  =  0.067,  partial  >f  =  0.115  (figure  1-21).  Participants  who  scored 
low  on  the  SpAO  measure  had  significantly  longer  glances  in  AOI  1 1  than  those  with  high  SpAO 
scores,  across  all  mission  conditions. 
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AOI 11  Average  Glance  Duration  by 
SpAO  Performance 


Figure  1-21.  AOI  11  average  glance  duration 
by  SpAO,  across  LOAs. 


1.14.6  Gaming  Experience 

There  were  no  significant  interactions  between  Gaming  Experience  and  LOA  on  Average 
Fixation  Duration,  Wilks’  X  =  0.951,  F  (2,  27)  =  0.690 ,/?  =  0.510,  partial  if  =  0.049,  or  Average 
Glance  Duration,  Wilks’  X  =  0.973,  F  (2,  27)  =  0.373,/?  =  0.692,  partial  rf  =  0.027. 

There  was  a  significant  between-subjects  effect  of  Gaming  Experience  on  Average  Fixation 
Duration,  F  { 1,  28)  =  4.409,/?  =  0.045,  partial  rf  =  0.136  (figure  1-22).  Average  Fixation 
Duration  was  significantly  shorter  for  Frequent  Action  Gamers  in  AOI  1 1  than  for  Other 
Gamers,  across  all  Mission  conditions.  There  was  no  significant  between-subjects  effect  of 
Gaming  Experience  on  Average  Glance  Duration,  A  (1,  28)  =  3.284,/?  =  0.081,  partial  tj  = 
0.105. 
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Figure  1-22.  AOI  1 1  average  fixation  duration  by 
gaming  experience,  across  LOAs. 


1.14.7  Clicking  Behavior 

Simple  Linear  Regression  showed  that  Clicking  behavior  in  AOI  1 1  was  predictive  of 
performance  on  the  Target  Detection  Task  for  the  Fully  Autonomous  condition  ((B  =  0.393,  t{ 27) 
=  2.265,  p  =  0.031),  but  not  for  the  Manual  ((3  =  0.264,  t(21)  =  1.146,/?  =  0.159)  or  Semi- 
Autonomous  ((3  =  0.206,  t(21)  =  1.010,/?  =  0.323)  conditions.  Clicking  Behavior  was  not 
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Correlated  with  any  individual  difference  factors.  The  fewest  participants  clicked  in  AOI  1 1  in 
the  Manual  condition  (66.7%),  while  the  RoboLeader  conditions  Semi- Autonomous  (83.3%)  and 
Fully  Autonomous  (76.7%)  had  more  usage  (figure  1-23). 


Figure  1-23.  Percent  of  participants  that  clicked  in  AOI  1 1  at  least  once, 
across  LOAs. 

1.15  AOIs  12, 13,  and  14  -  MGV  Rearward  180°  Camera  Feed 

AOIs  12,  13  and  14  comprise  the  MGV  rearward  180°  camera  feed,  and  collectively  account  for 
21.7%  of  the  screen  area.  Objects  appear  along  the  periphery  in  either  AOI  12  (on  the  right)  or 
AOI  14  (on  the  left)  before  disappearing  into  the  horizon  in  AOI  13  (straight  ahead).  The  views 
in  these  AOIs  could  not  be  manipulated  by  the  operator,  and  monitoring  these  feeds  was  essential 
for  identifying  threats  for  the  target  detection  task,  as  well  as  maintaining  SA.  It  was  expected 
that  AOI  13  would  have  more  fixations  and  glances  than  either  AOI  12  or  AOI  14,  primarily  due 
to  its  central  location  and  proximity  to  AOI  10.  Monitoring  these  feeds  was  necessary  for 
identifying  threats  for  the  target  detection  task,  as  some  threats  were  only  visible  in  this  rearward 
view.  As  such,  it  was  expected  that  participants  who  utilized  all  three  AOIs  for  the  target 
detection  task  would  have  higher  scores  and  fewer  FAs  than  those  who  did  not. 

1.16  AOI  12  MGV  Rearward  Right 

Simple  Linear  Regression  analysis  indicated  that  Fixation  Count,  Glance  Count,  and  Average 
Duration  of  Fixations  and  Glances  in  AOI  12  were  not  predictive  of  performance  on  the  Target 
Detection  Task  (table  1-23).  Simple  Linear  Regression  analysis  indicated  that  Glance  Count, 
Fixation  Count,  Average  Glance  Duration,  and  Average  Fixation  Duration  in  AOI  12  were  all 
predicative  of  the  number  of  FAs  reported  in  the  Manual  condition,  but  not  in  the  RoboLeader 
conditions  (table  1-24). 
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Table  1-23.  AOI  12  fixation  and  glance  behavior  for  target  detection  across  LOAs. 


Manual 

Semi 

Full 

Beta 

t 

Sig. 

Beta 

t 

Sig. 

Beta 

t 

Sig. 

Glance  Count 

1.645 

0.979 

0.338 

1.095 

0.794 

0.436 

-0.410 

-0.316 

0.755 

Avg  Glance  Duration 

0.680 

0.819 

0.421 

0.119 

0.186 

0.854 

-0.501 

-0.869 

0.394 

Fixation  Count 

1.251 

0.449 

0.657 

-3.801 

-1.576 

0.129 

1.772 

1.438 

0.164 

Avg  Fixation  Duration 

-0.066 

-0.096 

0.924 

-1.122 

-1.317 

0.201 

0.028 

0.060 

0.953 

Table  1-24.  AOI  12  fixation  and  glance  behavior  for  FAs  across  LOAs. 


Manual 

Semi 

Full 

Beta 

t 

Sig. 

Beta 

t 

Sig. 

Beta 

t 

Sig. 

Glance  Count 

4.892 

3.521 

0.002 

-0.352 

-0.191 

0.850 

-0.373 

-0.215 

0.832 

Avg  Glance  Duration 

1.778 

2.591 

0.017 

-0.216 

-0.253 

0.803 

0.343 

0.445 

0.660 

Fixation  Count 

-7.985 

-3.472 

0.002 

-0.610 

-0.190 

0.851 

-0.174 

-0.106 

0.917 

Avg  Fixation  Duration 

-1.059 

-1.864 

0.076 

0.156 

0.138 

0.892 

-0.507 

-0.801 

0.432 

1.16.1  LOA 

There  was  a  significant  main  effect  of  LOA  on  Fixation  Count,  Wilks’  X  =  0.694,  F  (2,  28)  = 

6.182 ,p  =  0.006,  partial  if  =  0.306,  (figure  I-24a),  as  well  as  on  Glance  Count,  Wilks’  X  =  0.738, 
F  (2,  28)  =  4.969,  p  =  0.014,  partial  rj2  =  0.262,  (figure  I-24b).  There  were  more  Fixations  and 
Glances  in  AOI  12  in  the  RoboLeader  conditions  than  in  the  Manual  condition. 
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Figure  1-24.  AOI  12  fixation  count  (a)  and  glance  count  (b),  across  LOAs. 
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There  was  a  marginally  significant  main  effect  of  LOA  on  Average  Glance  Duration,  Wilks’ 

X  =  0.815,  F  (2,  28)  =  3.812 ,p  =  0.057,  partial  if  =  0.185.  Average  Glance  Duration  in  AOI  12 
was  significantly  longer  in  the  Semi- Autonomous  condition  than  in  the  Manual  condition. 
Average  Glance  Duration  was  longer  in  the  Fully  Autonomous  condition  than  in  the  Manual 
condition,  but  did  not  reach  statistical  significance  (figure  1-25). 
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Figure  1-25.  AOI  12  average  glance  duration 
across  LOAs. 


1.16.2  Individual  Difference  Factors 

Average  Glance  Duration  in  AOI  12  was  positively  correlated  with  PAG  in  the  Manual  and 
Semi-Autonomous  conditions,  but  not  the  Fully  Autonomous  condition  (table  1-25).  Glance 
Count  and  Fixation  Count  in  AOI  12  were  negatively  correlated  with  SpAC  in  all  mission 
conditions  (table  1-26).  Glance  Count  and  Fixation  Count  in  AOI  12  were  moderately  correlated 
with  SpAO  in  the  Semi-Autonomous  condition,  but  not  the  Manual  or  Fully  Autonomous 
conditions  (table  1-27).  Gaming  Experience  was  not  correlated  with  Fixation  and  Glance 
behavior  in  AOI  12. 

Table  1-25.  AOI  12  usage:  PAC  correlations  with  fixation  and  glance  behavior  by  LOA. 


Attentional  Control 

Glances 

Count 

Total 

Glance 

Time 

Avg 

Glance 

Duration 

Fix 

Count 

Total 

Fixation 

Time 

Avg 

Fixation 

Duration 

Manual 

Pearson’s  r 

-0.213 

-0.022 

0.3943 

-0.069 

-0.014 

0.135 

Sig.  (2-tailed) 

0.258 

0.910 

0.031 

0.717 

0.940 

0.476 

Semi 

Pearson’s  r 

-0.130 

0.029 

0.384a 

0.007 

0.023 

0.164 

Sig.  (2 -tailed) 

0.493 

0.881 

0.036 

0.972 

0.904 

0.387 

Full 

Pearson’s  r 

-0.028 

0.091 

0.256 

0.022 

0.089 

0.294 

Sig.  (2-tailed) 

0.883 

0.633 

0.172 

0.908 

0.640 

0.114 

a  Correlation  is  significant  at  the  0.05  level  (2-tailed). 
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Table  1-26.  AOI  12  usage:  SpAC  correlations  with  fixation  and  glance  behavior  by  LOA. 


Spatial  Ability  Cube  Comparison  (SpAC) 


Glances 

Count 

Total 

Glance 

Time 

Avg 

Glance 

Duration 

Fix 

Count 

Total 

Fixation 

Time 

Avg 

Fixation 

Duration 

Manual 

Pearson’s  r 

-0.377“ 

-0.189 

0.211 

-0.312 

-0.156 

0.301 

Sig.  (2-tailed) 

0.040 

0.317 

0.263 

0.094 

0.411 

0.107 

Semi 

Pearson’s  r 

-0.403“ 

-0.317 

0.216 

-0.382“ 

-0.295 

0.231 

Sig.  (2-tailed) 

0.027 

0.087 

0.251 

0.037 

0.114 

0.219 

Full 

Pearson’s  r 

-0.467b 

-0.462“ 

-0.089 

-0.469b 

-0.437“ 

0.020 

Sig.  (2-tailed) 

0.009 

0.010 

0.641 

0.009 

0.016 

0.918 

a  Correlation  is  significant  at  the  0.05  level  (2-tailed). 
b  Correlation  is  significant  at  the  0.01  level  (2-tailed). 


Table  1-27.  AOI  12  usage:  SpAO  correlations  with  fixation  and  glance  behavior  by  LOA. 


Spatial  Ability  Orientation  (SpAO) 


Glances 

Count 

Total 

Glance 

Time 

Avg 

Glance 

Duration 

Fix 

Count 

Total 

Fixation 

Time 

Avg 

Fixation 

Duration 

Manual 

Pearson’s  r 

0.167 

0.065 

-0.253 

0.104 

0.049 

-0.087 

Sig.  (2-tailed) 

0.378 

0.732 

0.177 

0.585 

0.797 

0.649 

Semi 

Pearson’s  r 

0.340 

0.349 

0.030 

0.317 

0.333 

0.116 

Sig.  (2-tailed) 

0.066 

0.059 

0.874 

0.088 

0.072 

0.543 

Full 

Pearson’s  r 

0.222 

0.199 

-0.091 

0.180 

0.196 

-0.049 

Sig.  (2-tailed) 

0.239 

0.292 

0.634 

0.342 

0.300 

0.798 

1.16.3  PAC 

There  was  no  significant  interaction  between  PAC  and  LOA  on  Average  Glance  Duration, 

Wilks’  k  =  0.948,  F  (2,  27)  =  0.738, p  =  0.488,  partial  rj2  =  0.052. 

There  was  a  significant  between-subjects  effect  of  PAC  on  Average  Glance  Duration,  F  (1,  28)  = 
5.822,  p  =  0.023,  partial  rj  =  0.172  (figure  1-26).  Participants  with  higher  PAC  scores  had  longer 
Glances  in  AOI  12  than  those  with  lower  PAC  scores  across  all  mission  conditions. 
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AO  1 12  Average  Glance  Duration 
by  Attentional  Control 
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Figure  1-26.  AOI  12  average  glance  duration  by 
PAC,  across  LOAs. 


1.16.4  SpAC 

There  was  no  significant  interaction  between  SpAC  and  LOA  on  Fixation  Count,  Wilks’ 

X  =  0.944,  F  (2,  27)  =  0.806,  p  =  0.457,  partial  rf  =  0.056,  nor  between  SpAC  and  LOA  on 
Glance  Count,  Wilks’  X  =  0.969,  F  (2,  27)  =  0.433,  p  =  0.653,  partial  rj2  =  0.03 1 . 

There  was  a  significant  between-subjects  effect  of  SpAC  on  Fixation  Count,  F  (1,  28)  =  7.974, 
p  =  0.009,  partial  >f  =  0.222  (figure  I-27a),  and  Glance  Count,  F  (1,  28)  =  8.285,  p  =  0.008, 
partial  i /  =  0.228  (figure  I-27b).  Participants  with  lower  SpAC  scores  had  more  Fixations  and 
Glances  in  AOI  12  across  all  mission  conditions  than  those  with  higher  SpAC  scores,  and  had 
significantly  more  Fixations  and  Glances  in  the  RoboLeader  conditions  than  in  the  Manual 
condition.  Fixation  Count  and  Glance  Count  in  AOI  12  were  not  significantly  different  between 
mission  conditions  for  participants  with  high  SpAC  scores. 


Figure  1-27.  AOI  12  usage:  fixation  count  (a)  and  glance  count  (b)  by  SpAC,  across  LOAs. 
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1.16.5  SpAO 

There  was  no  significant  interaction  between  SpAO  and  LOA  on  Fixation  Count,  Wilks’  X  =  0.926, 
F  (2,  27)  =  1.072 ,p  =  0.356,  partial  rj2  =  0.074,  nor  between  SpAO  and  LOA  on  Glance  Count, 
Wilks’  X  =  0.931,  F  (2,  27)  =  0.994,  p  =  0.383,  partial  if  =  0.069.  There  was  not  a  significant 
between-subjects  effect  of  SpAO  on  Fixation  Count,  F(l,  28)  =  1.826,/?  =  0.187, 
partial  if  =  0.061,  or  Glance  Count,  F(l,  28)  =  2.449,  p  =  0.129,  partial  if  =  0.080  for  AOI  12. 

1.16.6  Clicking  Behavior 

Simple  Linear  Regression  indicated  that  Clicking  behavior  in  AOI  12  was  predictive  of 
performance  on  the  Target  Detection  Task  for  the  Fully  Autonomous,  P  =  0.351,  t{ 27)  =  1.983, 
p  =  0.057,  and  Semi-Autonomous,  p  =  0.373,  t{ 27)  =  2.125 ,p  =  0.043,  conditions,  but  not  for  the 
Manual  condition,  P  =  -0.047,  t{ 27)  =  -0.250 ,p  =  0.804.  Clicking  Behavior  in  AOI  12  was  not 
correlated  with  any  individual  difference  factors.  Overall,  62%  of  participants  did  click  in  AOI 
12,  with  this  percentage  being  lowest  in  the  Fully  Autonomous  condition  (50%)  and  higher  in  the 
Manual  (63%)  and  Semi-Autonomous  (73%)  conditions  (figure  1-28). 


Figure  1-28.  Percent  of  participants  that  clicked  in  AOI  12  at  least  once,  across 
LOAs. 

1.17  AOI  13  MGV  Rearward  Center 

Simple  Linear  Regression  analysis  indicated  that  Fixation  and  Glance  behavior  in  AOI  13  was 
not  predictive  of  performance  on  the  Target  Detection  Task  (table  1-28)  or  of  reported  FAs 
(table  1-29),  regardless  of  condition. 
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Table  1-28.  AOI  13  fixation  and  glance  behavior  for  target  detection  across  LOAs. 


Manual 

Semi 

Full 

Beta 

t 

Sig. 

Beta 

t 

Sig. 

Beta 

t 

Sig. 

Glance  Count 

-2.207 

-0.948 

0.354 

-1.422 

-0.586 

0.564 

1.235 

1.144 

Avg  Glance  Duration 

-0.618 

-0.719 

0.480 

-0.851 

-0.681 

0.503 

0.619 

0.963 

Fixation  Count 

7.583 

1.449 

0.161 

1.458 

0.435 

0.668 

-0.577 

-0.405 

Avg  Fixation  Duration 

0.756 

1.091 

0.287 

0.305 

0.213 

0.833 

-0.911 

-1.982 

Table  1-29.  AOI  13  fixation  and  glance  behavior  for  FAs  across  LOAs. 


Manual 

Semi 

Full 

Beta 

t 

Sig. 

Beta 

t 

Sig. 

Beta 

t 

Sig. 

Glance  Count 

0.395 

0.163 

0.872 

1.746 

0.636 

0.532 

-1.961 

-1.299 

0.207 

Avg  Glance  Duration 

-0.322 

-0.360 

0.722 

0.864 

0.611 

0.547 

-0.235 

-0.261 

0.796 

Fixation  Count 

-2.848 

-0.523 

0.606 

-3.908 

-1.030 

0.314 

2.269 

1.141 

0.266 

Avg  Fixation  Duration 

-0.068 

-0.094 

0.926 

-1.713 

-1.058 

0.302 

0.062 

0.096 

0.924 

1.17.1  LOA 

There  were  no  significant  main  effects  of  LOA  on  Fixation  Count,  Wilks’  X  =  0.974,  F  (2,  28)  = 
0.371,/?  =  0.693,  partial  rj2  =  0.026;  Glance  Count,  Wilks’  X  =  0.989,  F  (2,  28)  =  0.156,/?  = 
0.857,  partial  >f  =  0.01 1;  Average  Fixation  Duration,  Wilks’  X  =  0.903,  F  (2,  28)  =  1.508,/?  = 
0.239,  partial  rf  =  0.097;  or  Average  Glance  Duration,  Wilks’  X  =  0.936,  F  (2,  28)  =  0.953,/?  = 
0.398,  partial  tf  =  0.064  in  AOI  13. 

1.17.2  Individual  Difference  Factors 

Average  Glance  Duration  in  AOI  13  was  positively  correlated  with  PAC  in  the  Manual  and 
Semi-Autonomous  conditions,  but  not  the  Fully  Autonomous  condition  (table  1-30).  Glance 
Count  and  Fixation  Count  in  AOI  13  were  negatively  correlated  with  SpAC  scores  in  all  mission 
conditions  (table  1-31).  Glance  Count  and  Fixation  Count  in  AOI  13  were  moderately  correlated 
with  SpAO  in  the  Semi-Autonomous  condition,  but  not  the  Manual  or  Fully  Autonomous 
conditions  (table  1-32).  Gaming  Experience  was  not  correlated  with  Fixation  and  Glance 
behavior  in  AOI  13. 
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Table  1-30.  AOI  13  usage:  PAC  correlations  with  fixation  and  glance  behavior  by  LOA. 


Attentional  Control 

Glances 

Count 

Avg 

Glance 

Duration 

Fix 

Count 

Avg 

Fixation 

Duration 

Manual 

Pearson’s  r 

-0.213 

0.394a 

-0.069 

0.135 

Sig.  (2-tailed) 

0.258 

0.031 

0.717 

0.476 

Semi 

Pearson’s  r 

-0.130 

0.384a 

0.007 

0.164 

Sig.  (2-tailed) 

0.493 

0.036 

0.972 

0.387 

Full 

Pearson’s  r 

-0.028 

0.256 

0.022 

0.294 

Sig.  (2-tailed) 

0.883 

0.172 

0.908 

0.114 

a  Correlation  is  significant  at  the  0.05  level  (2-tailed). 


Table  1-31.  AOI  13  usage:  SpAC  correlations  with  fixation  and  glance  behavior,  by  LOA. 


Spatial  Ability  Cube  Comparison  (SpAC) 

Glances 

Count 

Avg 

Glance 

Duration 

Fix 

Count 

Avg 

Fixation 

Duration 

Manual 

Pearson’s  r 

-0.377“ 

0.211 

-0.312 

0.301 

Sig.  (2-tailed) 

0.040 

0.263 

0.094 

0.107 

Semi 

Pearson’s  r 

-0.403“ 

0.216 

-0.382“ 

0.231 

Sig.  (2-tailed) 

0.027 

0.251 

0.037 

0.219 

Full 

Pearson’s  r 

-0.467b 

-0.089 

-0.469b 

0.020 

Sig.  (2-tailed) 

0.009 

0.641 

0.009 

0.918 

a  Correlation  is  significant  at  the  0.05  level  (2-tailed). 
b  Correlation  is  significant  at  the  0.01  level  (2-tailed). 


Table  1-32.  AOI  13  usage:  SpAO  correlations  with  fixation  and  glance  behavior  by  LOAs. 


Spatial  Ability  Orientation  (SpAO) 

Glances 

Count 

Avg 

Glance 

Duration 

Fix 

Count 

Avg 

Fixation 

Duration 

Manual 

Pearson’s  r 

0.167 

-0.253 

0.104 

-0.087 

Sig.  (2-tailed) 

0.378 

0.177 

0.585 

0.649 

Semi 

Pearson’s  r 

0.340 

0.030 

0.317 

0.116 

Sig.  (2-tailed) 

0.066 

0.874 

0.088 

0.543 

Full 

Pearson’s  r 

0.222 

-0.091 

0.180 

-0.049 

Sig.  (2-tailed) 

0.239 

0.634 

0.342 

0.798 
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1.17.3  PAC 


There  was  no  significant  interaction  between  PAC  and  LOA  on  Average  Glance  Duration, 
Wilks’  X  =  0.883,  F  (2,  27)  =  1.787,  p  =  0.187,  partial  rj2  =  0.117.  There  was  no  significant 
between-subjects  effect  of  PAC  on  Average  Glance  Duration  for  AOI  13,  F  (1,  28)  =  0.497,/?  = 
0.487,  partial  if  =  0.017. 

1.17.4  SpAC 

There  were  no  significant  interactions  between  SpAC  and  LOA  on  Fixation  Count,  Wilks’  X  = 
0.924,  F  (2,  27)  =  1.118,/?  =  0.342,  partial  rj2  =  0.342,  or  between  SpAC  and  LOA  on  Glance 
Count,  Wilks’  X  =  0.940,  F  (2,  27)  =  0.865,/?  =  0.432,  partial  if  =  0.060. 

There  was  no  significant  between-subjects  effect  of  SpAC  on  Fixation  Count,  F  (1,  28)  =  3.23 1, 
p  =  0.083,  partial  if  =  0.103.  There  was  a  marginally  significant  between-subjects  effect  of 
SpAC  on  Glance  Count,  F  (2,  27)  =  3.694,  p  =  0.065,  partial  if  =  0.117,  (figure  1-29). 
Participants  with  lower  SpAC  scores  had  more  Glances  in  AOI  13  than  those  with  higher  SpAC 
scores  across  all  mission  conditions. 
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Figure  1-29.  AOI  13  usage:  glance  count  by  SpAC,  across 
LOAs. 


1.17.5  SpAO 

There  was  no  significant  interaction  between  SpAO  and  LOA  on  Fixation  Count,  Wilks’  X  = 
0.990,  F  (2,  27)  =  0.140,/?  =  0.870,  partial  rj2  =  0.010,  or  between  SpAO  and  LOA  on  Glance 
Count,  Wilks’  X  =  0.992,  F  (2,  27)  =  0.103,/?  =  0.902,  partial  rj2  =  0.008. 

There  were  no  significant  between-subjects  effect  of  SpAO  on  Fixation  Count,  F  (1,  28)  =  0.228, 
p  =  0.637,  partial  if  =  0.008,  or  Glance  Count,  F  (2,  27)  =  0.784,/?  =  0.383,  partial  if  =  0.027. 
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1.17.6  Clicking  Behavior 

All  participants  in  all  mission  conditions  clicked  in  AOI  13.  As  such,  Clicking  Behavior  in  AOI 
13  is  a  constant  and  could  not  be  analyzed  as  predictive  of  any  specific  outcomes. 

1.18  AOI  14  MGV  Rearward  Left 

Simple  Linear  Regression  analysis  indicated  that  Fixation  and  Glance  behavior  in  AOI  14  was 
not  predictive  of  performance  on  the  Target  Detection  Task  (table  1-33)  or  of  FAs  (table  1-34), 
regardless  of  condition. 

Table  1-33.  AOI  14  fixation  and  glance  behavior  for  target  detection  across  LOAs. 


Manual 

Semi 

Full 

Beta 

t 

Sig. 

Beta 

t 

Sig. 

Beta 

t 

Sig. 

Glance  Count 

-0.507 

-0.278 

0.783 

1.454 

1.038 

0.309 

-1.021 

-0.892 

0.381 

Avg  Glance  Duration 

-0.053 

-0.110 

0.913 

0.161 

0.227 

0.822 

-0.938 

-1.727 

0.096 

Fixation  Count 

0.855 

0.441 

0.663 

-1.256 

-0.902 

0.376 

1.300 

1.120 

0.273 

Avg  Fixation  Duration 

-0.096 

-0.213 

0.833 

-0.050 

-0.098 

0.923 

0.417 

0.992 

0.331 

Table  1-34.  AOI  14  fixation  and  glance  behavior  for  FAs  across  LOAs. 


Manual 

Semi 

Full 

Beta 

t 

Sig. 

Beta 

t 

Sig. 

Beta 

t 

Sig. 

Glance  Count 

-0.359 

-0.188 

0.853 

0.140 

0.088 

0.930 

0.323 

0.272 

0.788 

Avg  Glance  Duration 

-0.418 

-0.831 

0.414 

0.011 

0.014 

0.989 

0.714 

1.266 

0.217 

Fixation  Count 

0.519 

0.255 

0.801 

-0.115 

-0.073 

0.942 

-0.426 

-0.354 

0.727 

Avg  Fixation  Duration 

0.378 

0.799 

0.432 

-0.144 

-0.253 

0.803 

-0.555 

-1.272 

0.215 

1.18.1  LOA 

There  was  a  marginally  significant  main  effect  of  LOA  on  Fixation  Count,  Wilks’  X  =  0.825,  F 
(2,  28)  =  2.961,  p  =  0.068,  partial  if  =  0.175  (figure  I-30a),  and  on  Glance  Count,  Wilks’  X  = 
0.812,  F  (2,  28)  =  3.241,  p  =  0.054,  partial  if  =  0.188  (figure  I-30b).  Participants  made  fewer 
Fixations  and  Glances  in  AOI  14  in  the  Manual  condition  than  in  either  of  the  RoboLeader 
conditions. 
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Figure  1-30.  AOI  14  fixation  count  (a)  and  glance  count  (b),  across  LOAs. 

There  was  a  marginally  significant  main  effect  of  LOA  on  Average  Glance  Duration  in  AOI  14, 
Wilks’  X  =  0.822,  F  (2,  28)  =  3.040,  p  =  0.064,  partial  tf  =  0.178  (figure  1-31),  but  no  significant 
effect  on  Average  Fixation  Duration,  Wilks’  X  =  0.928,  F  (2,  28)  =  1 .087,  p  =  0.35 1,  partial 
rj2  =  0.072.  Average  Glance  Duration  in  AOI  14  was  significantly  shorter  in  the  Manual 
condition  than  in  the  Semi-Autonomous  or  Fully  Autonomous  conditions. 
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Figure  1-31.  AOI  14  average  glance  duration 
across  LOAs. 


1.18.2  Individual  Difference  Factors 

Fixation  Count,  Glance  Count  and  Average  Glance  Duration  in  AOI  14  were  negatively  correlated 
with  PAC  in  the  Manual  condition  (table  1-35),  but  not  in  either  RoboLeader  condition.  Average 
Glance  Duration  in  AOI  14  was  moderately  correlated  with  SpAC  in  the  Manual  condition 
(table  1-36),  but  not  in  either  Semi-Autonomous  or  Fully  Autonomous  conditions.  Fixation  Count 
and  Glance  Count  were  negatively  correlated  with  SpAC  in  the  Fully  Autonomous  condition,  but 
not  in  either  Manual  or  Semi- Autonomous  conditions.  Average  Glance  Duration  in  AOI  14  was 
negatively  correlated  with  SpAO  in  the  Fully  Autonomous  condition  (table  1-37),  but  not  in  either 
Manual  or  Semi-Autonomous  conditions.  Average  Fixation  Duration  and  Average  Glance 
Duration  in  AOI  14  were  negatively  correlated  with  Gaming  Experience  in  the  Manual  condition 
(table  1-38),  but  not  in  either  RoboLeader  condition. 
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Table  1-35.  AOI  14  usage:  PAC  correlations  with  fixation  and  glance  behavior 
by  LOA. 


Attentional  Control 

Glances 

Count 

Avg 

Glance 

Duration 

Fix 

Count 

Avg 

Fixation 

Duration 

Manual 

Pearson’s  r 

-0.377a 

-0.459a 

-0.40111 

-0.145 

Sig.  (2-tailed) 

0.040 

0.011 

0.028 

0.444 

Semi 

Pearson’s  r 

-0.043 

0.083 

0.000 

0.102 

Sig.  (2-tailed) 

0.823 

0.664 

1.000 

0.591 

Full 

Pearson’s  r 

-0.009 

0.172 

0.060 

0.111 

Sig.  (2-tailed) 

0.962 

0.364 

0.752 

0.560 

a  Correlation  is  significant  at  the  0.05  level  (2-tailed). 


Table  1-36.  AOI  14  usage:  SpAC  correlations  with  fixation  and  glance  behavior, 
by  LOA. 


Spatial  Ability  Cube  Comparison  (SpAC) 

Glances 

Count 

Avg 

Glance 

Duration 

Fix 

Count 

Avg 

Fixation 

Duration 

Manual 

Pearson’s  r 

-0.293 

0.331 

-0.273 

0.293 

Sig.  (2-tailed) 

0.117 

0.074 

0.145 

0.116 

Semi 

Pearson’s  r 

-0.111 

0.242 

-0.067 

0.163 

Sig.  (2-tailed) 

0.559 

0.198 

0.724 

0.388 

Full 

Pearson’s  r 

-0.444a 

-0.031 

-0.4223 

-0.021 

Sig.  (2-tailed) 

0.014 

0.870 

0.020 

0.914 

a  Correlation  is  significant  at  the  0.05  level  (2-tailed). 


Table  1-37.  AOI  14  usage:  SpAO  correlations  with  fixation  and  glance  behavior 
by  LOAs. 


Spatial  Ability  Orientation  (SpAO) 

Glances 

Count 

Avg 

Glance 

Duration 

Fix 

Count 

Avg 

Fixation 

Duration 

Manual 

Pearson’s  r 

0.235 

-0.244 

0.141 

0.026 

Sig.  (2-tailed) 

0.212 

0.193 

0.456 

0.892 

Semi 

Pearson’s  r 

0.184 

-0.249 

0.042 

0.091 

Sig.  (2-tailed) 

0.331 

0.185 

0.825 

0.633 

Full 

Pearson’s  r 

0.142 

-0.368a 

-0.006 

-0.039 

Sig.  (2-tailed) 

0.455 

0.045 

0.973 

0.838 

a  Correlation  is  significant  at  the  0.05  level  (2-tailed). 
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Table  1-38.  AOI  14  usage:  gaming  experience  correlations  with  fixation  and 
glance  behavior  by  LOA. 


Gaming  Experience 

Glances 

Count 

Avg 

Glance 

Duration 

Fix 

Count 

Avg 

Fixation 

Duration 

Manual 

Pearson’s  r 

-0.010 

-0.364“ 

-0.032 

-0.378a 

Sig.  (2-tailed) 

0.957 

0.048 

0.866 

0.039 

Semi 

Pearson’s  r 

-0.064 

-0.290 

-0.135 

-0.140 

Sig.  (2-tailed) 

0.736 

0.120 

0.477 

0.459 

Full 

Pearson’s  r 

0.278 

-0.195 

0.222 

-0.173 

Sig.  (2-tailed) 

0.137 

0.302 

0.239 

0.361 

Correlation  is  significant  at  the  0.05  level  (2-tailed). 


1.18.3  PAC 


There  were  significant  interactions  between  PAC  and  LOA  on  Fixation  Count  in  AOI  14,  Wilks’ 
X  =  0.741,  F  (2,  27)  =  A. 121,  p  =  0.017,  partial  if  =  0.259  (figure  I-32a),  and  between  PAC  and 
LOA  on  Glance  Count  in  AOI  14,  Wilks’  X  =  0.748,  F  (2,  27)  =  4.543,  p  =  0.020,  partial  rj2  = 
0.252  (figure  I-32b).  Participants  with  high  PAC  had  significantly  fewer  Fixations  and  Glances 
in  Manual  condition  than  those  with  low  PAC,  but  roughly  the  same  amount  of  Fixations  and 
Glances  as  those  with  low  PAC  in  the  Semi- Autonomous  and  Fully  Autonomous  conditions. 


(a) 


AOI  14  Glance  Count  by 
Attentional  Control 


(b) 


Figure  1-32.  AOI  14  fixation  count  (a)  and  glance  count  (b)  by  PAC  (PAC),  across  LOAs. 

There  was  a  significant  interaction  between  PAC  and  LOA  on  Average  Glance  Duration  in  AOI 
14,  Wilks’  X  =  0.755,  F  (2,  27)  =  4.388,/?  =  0.022,  partial  rj2  =  0.245  (figure  1-33).  Participants 
with  high  PAC  had  significantly  shorter  Glances  in  Manual  condition  than  those  with  low  PAC, 
but  roughly  the  same  length  of  Glances,  as  those  with  low  PAC  in  the  Semi-Autonomous  and 
Fully  Autonomous  conditions. 


104 


AOI 14  Average  Glance  Duration 
by  Attentional  Control 
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Figure  1-33.  AOI  14  average  glance  duration  by  PAC, 
across  LOAs. 


1.18.4  SpAC 

There  was  no  significant  interaction  between  SpAC  and  LOA  on  Fixation  Count  in  AOI  14, 
Wilks’  X  =  0.899,  F  (2,  27)  =  1.509 ,/?  =  0.239,  partial  rj~  =  0.101,  or  between  SpAC  and  LOA  on 
Glance  Count  in  AOI  14,  Wilks’  X  =  0.891,  F  (2,  27)  =  1.651,/?  =  0.21 1,  partial  rj2  =  0.109. 

There  was  no  significant  interaction  between  SpAC  and  LOA  on  Average  Glance  Duration  in 
AOI  14,  Wilks’  X  =  0.874,  F  (2,  27)  =  1.954,/?  =  0.161,  partial  rj2  =  0.126.  There  was  no 
significant  between-subjects  effect  of  SpAC  on  Average  Glance  Duration  in  AOI  14,  F  (1,  28)  = 
1.364,/?  =  0.253,  partial  tj  =  0.046.  There  were  marginally  significant  between-subjects  effects 
of  SpAC  on  Fixation  Count,  F(l,  28)  =  3.905,/?  =  0.058,  partial  if  =  0.122  (figure  I-34a),  and 
on  Glance  Countin  AOI  14,  F(l,  28)  =  4.121,/?  =  0.052,  partial  if  =  0.128  (figure  I-34b). 
Participants  who  scored  high  in  SpAC  had  significantly  fewer  Fixations  and  Glances  in  AOI  14 
than  those  with  low  SpAC  scores  across  all  mission  conditions. 


Figure  1-34.  AOI  14  fixation  count  (a),  and  glance  count  (b)  by  SpAC,  across  LOAs. 
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1.18.5  SpAO 


There  was  no  significant  interaction  between  SpAO  and  LOA  on  Average  Glance  Duration, 
Wilks’  X  =  0.965,  F  (2,  27)  =  0.483,/?  =  0.622,  partial  rj2  =  0.035.  There  was  a  significant 
between-subjects  effect  of  SpAO  on  Average  Glance  Duration  in  AOI  14,  F  (1,  28)  =  4.442 ,p  = 
0.044,  partial  rf  =  0.137  (figure  1-35).  Participants  who  scored  high  in  SpAO  had  significantly 
shorter  Glances  in  AOI  14  than  those  with  low  SpAO  scores  across  all  mission  conditions. 
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Figure  1-35.  AOI  14  average  glance  duration  by 
SpAO,  across  LOAs. 


1.18.6  Gaming  Experience 

There  was  no  significant  interaction  between  Gaming  Experience  and  LOA  on  Average  Fixation 
Duration,  Wilks’  X  =  0.897,  F  (2,  27)  =  1.548,/?  =  0.231,  partial  if  =  0.103,  or  between  Gaming 
Experience  and  LOA  on  Average  Glance  Duration,  Wilks’  X  =  0.987,  F  (2,  27)  =  0.177,/?  = 
0.839,  partial  rj2  =  0.013. 

There  was  a  marginally  significant  between-subjects  effect  of  Gaming  Experience  on  Average 
Glance  Duration  in  AOI  14,  F  (1,  28)  =  3.985,/?  =  0.056,  partial  ij2  =  0.125  (figure  1-36),  but  not 
on  Average  Fixation  Duration,  F(l,  28)  =  1.872,/?  =  0. 1 82,  partial  rj  =  0.063.  Frequent  Action 
Gamers  had  significantly  shorter  Glances  in  AOI  14  than  All  Other  Gamers  across  all  mission 
conditions. 
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Figure  1-36.  AOI  14  average  glance  duration  by 
gaming  experience,  across  LOAs. 


1.18.7  Clicking  Behavior 


Clicking  Behavior  in  AOI  14  was  not  correlated  with  performance  on  the  Target  Detection  Task 
or  reported  FAs  in  any  Mission  condition,  nor  was  Clicking  Behavior  correlated  with  any 
individual  difference  measures.  Overall,  64%  of  participants  did  click  in  AOI  14,  with  this 
percentage  being  lowest  in  the  Manual  and  Semi-Autonomous  conditions  (60%),  and  highest  in 
the  Fully  Autonomous  condition  (73.3%),  (figure  1-37). 
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Figure  1-37.  Percent  of  participants  that  clicked  in  AOI  14  at  least  once,  across 
LOAs. 
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Intentionally  Left  Blank. 
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Appendix  J.  Participant  Comments  from  the  Usability  and  Trust  Survey 


This  appendix  appears  in  its  original  form,  without  editorial  change. 
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Participant  comments  as  written  on  the  Usability  and  Trust  survey,  comments  section: 

•  “Provide  zoom  capability  for  UGV  camera  so  the  user  could  make  better  use  of  it.” 

•  “Would  like  auditory  feedback  when  clicking  on  threats” 

•  “UAS  was  not  useful.  UGV  was  a  little  useful,  but  not  much.  Pace  was  slow  enough  for 
just  the  MGV  [camera  feeds].” 

•  “Threats  were  visible  from  a  longer  distance,  but  not  clearly  rendered  and  easily 
identifiable.” 

•  “UAS  felt  useless,  MGV  too  warped  in  the  periphery.  UGV  was  great,  RoboLeader  was 
great.” 

•  Waypoints  on  top  of  one  another  could  not  be  clicked  on  separately. 

•  “Hardly  used  UAS,  but  UGV  and  MGV  were  useful” 

•  To  the  question  ‘The  RoboLeader  system  has  integrity’,  “I’m  not  sure  until  that  can  be 
proven  over  time.” 

•  “Allowing  the  human  id  targets  rather  than  id’ing  and  path  planning  would  add  a  great  deal 
of  security  to  a  mission.” 

•  To  the  question  ‘The  RoboLeader  system  may  have  harmful  effects  on  the  task,’  “I  feel  it 
could  with  terrain  choices,  but  I  personally  never  had  to  reroute  it.”  “Taking  too  much 
control  away  (i.e.  routing,  spacing)  leaves  the  user  less  spatially  aware  and  more  concerned 
with  flagging  threats  which  can  be  harmful  depending  on  the  priorities  of  the  task.” 
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List  of  Symbols,  Abbreviations,  and  Acronyms 


ANOVA 

analysis  of  variance 

AOI 

area  of  interest 

DV 

dependent  variable 

FA 

false  alann 

FPS 

first-person  shooter 

GE 

gaming  experience 

HRI 

human-robot  interaction 

LOA 

level  of  autonomy 

MGV 

manned  ground  vehicle 

MIX 

mixed  initiative  experimental  testbed 

NASA 

National  Aeronautics  and  Space  Administration 

ocu 

operator  control  unit 

PAG 

perceived  attentional  control 

RED 

remote  eyetracking  device 

SA 

situation  awareness 

SMI 

Sensomotoric  Instrument 

SpA 

spatial  ability 

SpAC 

spatial  ability  -  cube  comparison  test 

SpAO 

spatial  ability  -  spatial  orientation  test 

UAS 

unmanned  aerial  system 

UGV 

unmanned  ground  vehicle 

111 


NO.  OF 

COPIES  ORGANIZATION 


NO.  OF 

COPIES  ORGANIZATION 


1  DEFENSE  TECHNICAL 
(PDF)  INFORMATION  CTR 
DTIC  OCA 

1  DIRECTOR 

(PDF)  US  ARMY  RESEARCH  LAB 
IMAL  HRA 

1  DIRECTOR 

(PDF)  US  ARMY  RESEARCH  LAB 
RDRL  CIO  LL 

1  GOVT  PRINTG  OFC 
(PDF)  AMALHOTRA 

1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  RDRL  HRM  C  A  DAVISON 
320  MANSCEN  LOOP  STE  115 
FORT  LEONARD  WOOD  MO  65473 

1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  RDRL  HRM  D 
T  DAVIS 

BLDG  5400  RM  C242 

REDSTONE  ARSENAL  AL  35898-7290 

1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  RDRL  HRS  EA  DR  V  J  RICE 
BLDG  4011  RM  217 
1750  GREELEY  RD 

FORT  SAM  HOUSTON  TX  78234-5002 

1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  RDRL  HRM  DG  J  RUBINSTEIN 
BLDG  333 

PICATINNY  ARSENAL  NJ  07806-5000 

1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  ARMC  FIELD  ELEMENT 

RDRL  HRM  CH  C  BURNS 
THIRD  AVE  BLDG  1467B  RM  336 
FORT  KNOX  KY  40121 

1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  AWC  FIELD  ELEMENT 

RDRL  HRM  DJ  D  DURBIN 
BLDG  4506  (DCD)  RM  107 
FORT  RUCKER  AL  36362-5000 

1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  RDRL  HRM  CK  J  REINHART 
10125  KINGMAN  RD  BLDG  317 
FORT  BELVOIR  VA  22060-5828 


1  ARMY  RSCH  LABORATORY  -  HRED 

(PDF)  RDRL  HRM  AY  M  BARNES 
2520  HEALY  AVE 
STE  1172  BLDG  51005 
FORT  HUACHUCA  AZ  85613-7069 

1  ARMY  RSCH  LABORATORY  -  HRED 

(PDF)  RDRL  HRM  AP  D  UNGV ARSKY 
POPE  HALL  BLDG  470 
BCBL  806  HARRISON  DR 
FORT  LEAVENWORTH  KS  66027-2302 

1  ARMY  RSCH  LABORATORY  -  HRED 

(PDF)  RDRL  HRM  AT  J  CHEN 
12423  RESEARCH  PKWY 
ORLANDO  FL  32826-3276 

1  ARMY  RSCH  LABORATORY  -  HRED 

(PDF)  RDRL  HRM  AT  J  WRIGHT 
12423  RESEARCH  PKWY 
ORLANDO  FL  32826-3276 

1  ARMY  RSCH  LABORATORY  -  HRED 

(PDF)  RDRL  HRM  AT  S  QUINN 
12423  RESEARCH  PKWY 
ORLANDO  FL  32826-3276 


1  ARMY  RSCH  LABORATORY  -  HRED 

(PDF)  RDRL  HRM  AT  C  KORTENHAUS 
12350  RESEARCH  PKWY 
ORLANDO  FL  32826-3276 

1  ARMY  RSCH  LABORATORY  -  HRED 

(PDF)  RDRL  HRM  CU  B  LUTAS-SPENCER 
6501  E  11  MILERD  MS  284 
BLDG200A  2ND  FL  RM2104 
WARREN  MI  48397-5000 

1  ARMY  RSCH  LABORATORY  -  HRED 

(PDF)  FIRES  CTR  OF  EXCELLENCE 
FIELD  ELEMENT 
RDRL  HRM  AF  C  HERNANDEZ 
3040  NW  AUSTIN  RD  RM  221 
FORT  SILL  OK  73503-9043 

1  ARMY  RSCH  LABORATORY  -  HRED 

(PDF)  RDRL  HRM  AV  W  CULBERTSON 
91012  STATION  AVE 
FORT  HOOD  TX  76544-5073 
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NO.  OF  NO.  OF 

COPIES  ORGANIZATION  COPIES 

1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  HUMAN  RSRCH  AND  ENGRNG 

DIRCTRT  MCOE  FIELD  ELEMENT 
RDRL  HRM  DW  C  CARSTENS 
6450  WAY  ST 
BLDG  2839  RM  310 
FORT  BENNING  GA  31905-5400 

1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  RDRL  HRM  DE  A  MARES 

1733  PLEASONTONRD  BOX  3 
FORT  BLISS  TX  79916-6816 

8  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  SIMULATION  &  TRAINING 
TECHNOLOGY  CENTER 
RDRL  HRT  COL  M  CLARKE 
RDRL  HRT  I  MARTINEZ 
RDRL  HRT  T  R  SOTTILARE 
RDRL  HRT  B  N  FINKELSTEIN 
RDRL  HRT  G  A  RODRIGUEZ 
RDRL  HRT  I  J  HART 
RDRL  HRT  M  C  METEVIER 
RDRL  HRT  S  B  PETTIT 
12423  RESEARCH  PARKWAY 
ORLANDO  FL  32826 

1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  HQ  USASOC 

RDRL  HRM  CN  R  SPENCER 
BLDG  E2929  DESERT  STORM  DRIVE 
FORT  BRAGG  NC  28310 

1  ARMY  G1 
(PDF)  DAPE  MR  B  KNAPP 

300  ARMY  PENTAGON  RM  2C489 
WASHINGTON  DC  20310-0300 


ABERDEEN  PROVING  GROUND 

12  DIRUSARL 
(PDF)  RDRL  HR 

L  ALLENDER 
P  FRANASZCZUK 
C  COSENZO 
RDRL  HRM 

P  SAVAGE-KNEPSHIELD 
RDRL  HRM  AL 
C  PAULILLO 
RDRL  HRM  B 
C  SAMMS 
RDRL  HRM  C 
L  GARRETT 


ORGANIZATION 

RDRL  HRS 
J  LOCKETT 
RDRL  HRS  B 
M  LAFIANDRA 
RDRL  HRS  C 
K  MCDOWELL 
RDRL  HRS  D 
B  AMREIN 
RDRL  HRS  E 
D  HEADLEY 
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